Databricks Cost Tracking: FinOps Tools & Best Practices

0:04today we have a special guest we have the famous Greg from The Money Team and of course Quinton hi Greg hey thanks for having me can you introduce yourself absolutely uh my name is Greg I'm one of the product managers here at Databricks I I work with um The Money Team which is our our fun name for the team internally

0:27that handles a lot of things related to billing cost management observability of usage and what's happening and so I spend a lot of time working on features that help customers understand what's going on uh with Databricks their instance of it how uh usage is going the health of things Etc that's very very interesting and I I think observability and phobs are very

0:50important topic and can you like work us through like the evolution of the uh this observability uh let's say Journey that started I think two years ago yeah absolutely so you're right uh this is an exciting area what I tend to see is that for customers as they spend more money it becomes more important to have

1:13observability and put finops best practices in place um when you start off um you know you're maybe exploring some data sets trying out the product um you often don't see as much focus on that there's more focus on sort of using the product and does this meet my needs and is it fast but over time as things SK scale and you start to have tens or hundreds or thousands of data workers and a lot of data being managed you want

1:35to understand you know what am I spending money on is it effective and so we see a lot of that um you know as customers ramp up and grow um and over the last couple of years we've been investing a lot to make sure that we have the information they need ready for them we took a data first approach not a big surprise we're a data company and so

1:57um the first thing we said is let's make it really easy easy to get all data about what you're using and we built system tables which was a a great way to use sort of native within Databricks and get access to data we've seen a lot of people do cool things off of that whether it's plug it into existing tools or dashboards they had use it you know natively at Databricks or integrate that data with information from their

2:19cloud provider or other sources um we've then started investing in tooling to make it easier and kind of a little less do it yourself say hey let's let you just go ahead and do these common scenarios you want to do like set up a budget alert or drill down into dashboards um and then we've been continuing to invest in uh a number of things this year we've spent a lot of time helping folks drill down to kind of

2:44more granular aspects of their spending what started as hey I want to see my high level spend then turned into well I want to break it down by team I have a few different projects or teams and I want to see this spend and then that next level of granularity is actually I want to see the individual things happening the job runs or the queries and I want to understand those because that tends to be where you can actually

3:07take action and make a change and optimize something or coach a data worker on like how to better use the partitions that are available so they're not scanning so much data and the query is faster so we've really kind of gone from that high level to much more grar we've made a lot of progress there over this last year which has been exciting yeah and um I remember like for system tables before we used to have uh I think

3:30was called audit logs that you needed like to activate this then have like a bunch of Json files that you had to like par and then make them available in a table now it's super easy just go enable system tables you and have a bunch of schemas including the one on the building the uh the usage then you can you can like build a lot of

3:53dashboards and I've seen like for the last maybe six months we did accelerate a lot in this phobs part uh like like by like releasing stuff on tracking cost First for for serverless then we added the model survey like every like talking about workflows uh

4:15notebooks then model serving we're adding we are adding more and more also like some of the on uh pre-built dashboards so are we planning like to focus more on uh bringing in more dashboards or like enhancing the exist exists in on or yeah so we've definitely taken an approach where we've said every product on Databricks should uh be easy

4:39for customers to understand their cost and their spending to put tags in place so they can do cost attribution and see like which spend is happening by which team um now we've expanded a lot as a company even in the few years since I've been here we've gone from a couple products to a lot more and individual features and part of um allowing customers to understand their spend is

5:01doing it consistently it does you no good if you can understand half your spend but you can't understand the other half and so part of what we've been doing is trying to kind of raise the floor on all products um in terms of the finops that's available and make sure that they can put tagging in place they can see that span that it shows up in those cost dashboards and so a lot of things have happened on um yeah the AI

5:24ml space um serverless products um to help that um there's also been a lot happening on some of our our more classic products um the lake flow team has done a lot to release more data on jobs and job runs um and you know letting people see detail the SQL team has done a lot with queries the query history table letting you see individual

5:46queries and how long they take um see the warehouses and information about them and so that really lets you drill in a bit further um we still have some products that have a little bit of catching up to do um but we're getting to that point where they're going to cat catch up and then all new products we release in the future will really have a certain bar they have to meet and so that's one of the things our team is trying to do is build a platform that

6:09makes it easy for when a new product is built it can launch with all that cost management built in so there's not sort of that weight to say Well when is this stuff going to come it's just kind of out of the box on day one and that's part of our maturing Journey as a company um you know you start off as a company and you build a great product and you're really excited people are using it and then they start saying like hey I want to see where I'm spending eventually you get to the point where

6:31you're like cool this stuff is all built in as we launch features you know there's checkboxes for all the types of things certainly finop certainly security networking compliance type things um Regional availability all those things start to become standard as you mature in a company and we've been doing that over the last few years here at data Brooks yeah and I think you so you have access as as an account mean

6:53you can get out of the bug dashboard from the AC UI and I'm going to add like a sham plug but on top of that you can also install all the DB demo system tables that we have and we try to so it's more like Community Driven uh but we have a lot of dashboards on everything um Warehouse model SQL model serving endpoint sorry and you can really go into the details and see you know which box space is consuming what

7:16and you can zoom in and get really lot of information so we try to pre-bake all these dashboards within the DB demo system table one um so like it's super useful uh and just before like Quinton asked the question in case you want to use the dashboard you mention like the budget like the uh budget tracking need to make sure that you enable uh system tables activate

7:40them and maybe refresh uh like the uh the schemas maybe every quarter to have the latest tables which are used to uh to feed the dashboard that you're going to be using yeah absolutely um the good news is our GA tables the billing tables are automatically enabled and so kind of no need for that refresh there but we do have a number of tables in preview that yeah you can enable and make sure you

8:03have all the latest things so you can use some of those dashboards the Community Driven thing I think is really important we took this approach of let's give customers data about what's happening let's give them some sample queries and kind of see what they can do with it and it wasn't just customers it was customers and it was internal teams and um you know partnership on that and people have published blog posts and

8:25some Databricks Partners have built things on Papa system tables and we like that because you're going to have more Innovation you're going to have cool things come out of that rather than us just saying here's like one report we built and there's a chart and you can't do anything with it that data first approach gives you some more openness now the challenge of that is there's a bunch of these dashboards floating around and you have one that does this

8:47and one that does that and there's some questions about it and so balancing those two things is something we've been thinking about how do we make sure we give people an official blessed dashboard where they can trust the numbers but also give them access to some of this sort of like community-driven stuff and it's it's something I've been noodling on um because in the area observability um

9:09yeah there's just always going to be new ways to think about things that ask things and we don't want to limit that but at the same time if it it's the two spread out there's questions about like hey these numbers don't match or what does this one mean or which one of these should I use and so I think there is a balance between those things and um yeah I suspect that scenario we'll be talking more about in the future so maybe what advice would you share with someone who

9:33you know want to implement best practices I feel like there is a notion of observability so you want to know what's going on but potentially also you want to put you know budget in place alert quite yeah Best Practices you would recommend to a new user often what I see happen with customers is they start off using Databricks at a small scale and then they're really excited about the product and the features and things grow and they add more users and

9:57then they sort of have a a wo moment where they realize they're kind of spending a little more money than they thought or they have a question from their CFO or something about like hey tell me how much this team spent and then all of a sudden they realize like okay I need to like break down the spend and I have three different projects and they need to understand which is you know how much is being spent on this or that project some

10:21companies get there pretty quickly some take a little while to ramp up and get there but I feel like everyone kind of has that moment and if you wait until that moment to think think about cost stuffff you're you're sort of too late now you have all this usage that maybe is intertangled and it's hard to understand and so what I always recommend is as people get started to organize Databricks in a way that'll

10:43make it easy in the future and there's a couple ways you can do that one is by setting up different workspaces and kind of isolating teams and so you have things divided all the system tables have workspace ID as a column and so it's easy to split things that way um another you can do is set up different shared compute resources if you have people on all-purpose compute for example um have one team kind of sharing

11:06that cluster and then you'll know all the spend on it so that team now another thing you can do is set up tagging and you can use the same tags across different things you can tag on workspaces or warehouses or clusters um and and you know more and more features are adding places where you can add tags and that will let you then just be able to say well I want to see the sum of costs for this tag and now you can see

11:28it across everything so usually it's a combination of those things and it's hard for me to say one way to do it because every company's different um in how they organize themselves and how they want to break down costs but it's usually a combination of those and if you set that up in your early days you're going to have a much better time down the road when you want to break down your costs because everything is segmented out it

11:51it's also almost impossible to tag too much like it can become a bit of an overhead if you have a lot of things and it's a little confusing but like if you have too many tags you can always ignore one of them in the query later and do the reporting but if you don't have enough tags you're out of luck you have no way of splitting things or you have to you know try to figure something out after the fact and so putting some tags

12:14in place at the beginning is great if you decide later you don't like those tags you can always add new ones and you just kind of like start ignoring those and so yeah that's always day one get started by putting a little bit of organization in place in sort of a logical way um and most people are famili with this cuz if they're using other tools if they're using a cloud provider they've set up this stuff I think it's just realizing hey I should do this on Databricks as well and I

12:37remember like what I was sharing with my customers and thanks to the feature like released uh budget policies a lot of things like was much easier because I remember in the past I used to push them to use cluster policies and by team why because like the tagging was unforced automatically like if you are part of data engineering team then you'll have

12:59when whenever you create a cluster is going to be tagged I don't know team data engineering same thing data analyst ml team and then with the sist offering with the budget policies you you can do the same thing if this person is using several notebooks the C will be Associated to data engineering team if it's workflow same thing and model suring as well and I think I don't know

13:23if I'm supposed to say this but DBC like SQL warehouses this is something that will come little bit later which means all the resources are now tacked and then you can just go and deploy the uh the uh dashboard to track the consumption you can also use the alerts that your team also released like just select the alerts maybe the guys behind

13:46the phobs put a budget and if they are above the threshold They will receive an alert and this is this is super cool and I think this is something you're planning to demo maybe for us yeah I can show that off um definitely we can we can hop into a demo but I think you're right that's um being able to put those tags on everything um so the budget policies

14:10which allows the tagging and it it does it in a different way which is kind of interesting um some of our Databricks products that have been around a while use shared compute and you would tag the compute and so you kind of had a limit to the granularity you could tag at and you know you would have to say like oh I want another cluster if I want to have a different tag as we're building serverless we sort of get to think about things again and say hey how would we

14:33want to do this now and one of the things we've done is we've made it so that everything that kicks off um basically an execution on server list can be individually tagged and it's individually reported in the usage records and so that gives you a much finer grain thing to operate on and you can actually have 50 people each with a different tag um whereas before those 50

14:57people might have been on the same cluster and so had the same tag so it's much easier to break it up into smaller things and then group them later and so we do see people doing that where they'll put um that policy in place they'll tag people based on their team or something and then they'll maybe join that with an org chart table that they've imported and say okay now we can group it by team or by project that

15:19they're on um or they can set up some of those tags um by putting users and groups um with those projects if if it's relatively fixed and so this is an ER where I'd say we're able to do more on server list because we're kind of building fresh and restarting whereas doing some of that um on classic would take a lot of rework and it's really hard to change some of those things once you've made some of those decisions and

15:42so it is exciting to see how how fine grain you can get on newer things whether it's model serving whether it's uh you know serverless interactive Etc so let's see I'm happy to do a demo um how do I share my screen here let's see and I have one one one more question for you because I've seen we are currently like building more dashboards I've seen the one on the workflows and are we intending at the

16:06end like uh and also the one on DBQ are we intending to uh share like or have like one uh let's say dashboard with multi Pages where we're going to have all this observ stuff or we still going to like maybe like try to unify them or what's what's the uh the goal yeah absolutely so when we built the cost

16:28dashboard which is sort of the Databricks blessed here's the best way to think about looking at your costs we said let's start with the high level spend and so we have here you can see your spend over time we're looking at month and we have it by certain product line categories whether it's jobs or DLT Etc um and then we let customers start drilling down you I mentioned previously

16:50tagging was important because you wanted to see teams spend and so you can look and say okay I have a couple different teams how much is each of those teams spending over time um and then we went a step further into like well when we have various metadata on them you can drill in like endpoint name for the things that have it which not everything is an endpoint because obviously stuff that's jobs isn't an endpoint and won't have an

17:13endpoint name but for the end points you could start to see that one of the next things that came up is well for SQL specifically I want to drill down and look at this and for L flow I want to drill down and look at this stuff and the products are a little different so the questions that you ask are a little different and the way we've been doing that is each of those product teams is investing in answering the core questions and I've

17:35been working with the PMs and Engineers on each of those teams to say okay when you're answering those questions let's make sure there's a natural flow from the centralized cost dashboard that shows you your overall spend down to what you're showing so you can kind of drill down and then The Logical way would be to group those together now the great news is as we build these dashboards the AI team has been building

17:58a lot of things um the ability to have kind of more complex dashboards have tabs Etc and so that's one of the angles we're exploring is could this just be you start at the center and then when you want to drill down further you know we have multiple tabs or a separate tab over there um we've been discussing other options as well um we're continuing to explore um what is the right way to get this data

18:21to customers the is it a native aibi dashboard that we've sort of pre-installed is it something that's ented that they can just see and use and so there's a couple different methods to it um but you know we always like hearing what customers are using um and part of the benefit of launching these kind of specific dashboards for each product is we get to really get that

18:43customer validation and make sure that one answers their questions and then we can figure out how to pull it into the fold and so there's a little bit I mentioned earlier this notion of Community Driven we see a lot of people doing development and then we want to take the best IDE is and sort of make them the gold standard and so some of that is even development happening within data birks we want that team to

19:05go be able to experiment and feel free to iterate quickly and learn and then once they've really nailed it we can say great now let's make that a part of the gold standard so that's an area that we'll definitely be having some more announcements soon in that's great and Greg I know I know a lot of folks don't know about the like the policy they can do and the tagging they can add on the policy is that something you can you can show us to do you have that available

19:28let's look at policies so policies are a great way uh to do a number of things in Databricks specifically with serverless you can use a feature we have called budget policies which helps you with attribution and tagging so if we go into our workspace settings and compute here we can see budget policies so we can create one of these to apply a tag um

19:51let's say we want to call this one Money Team and I want everything uh that we do on The Money Team to be Tagged so that we can uh you know report up to our CFO what we're spending we can go um so this is going to be key value pairs so we could do um budget and then Money Team as our tag and then when I use something

20:15and I'm using this policy that tag is going to apply so we're going to create it um that is now live and what we are going to do is I am a manager we're going to create a new notebook um and we are going to make sure um I am using serverless

20:37compute um so I am set up now in this case I only have one policy assigned to myself and so it is going to default to using that policy in the case that there were multiple policies for me um such as if I worked on two different projects and needed to periodically assign the cost I could switch between them

20:59um in this case I am uh going to be using that policy by default there we go um perfect budget policy Money Team and then we're going to tag uh with uh Money Team so all this cost usage is going to get tagged there so that's where we can see it if we had multiple we can switch between them um in this case uh we have

21:24have some tests that I'm not actually or or maybe they're assigned to everyone and so that's why um now that information that tagging information is going to flow all the way through to system tables um and so you can you know run queries on that you can Group by it and so you have those tags to do various things our uh aibi cost daspoort utilizes those tags so you can

21:46break down costs by spend easily and then you can also set up budget alerts using those tags and so U there's a number of ways to set up budget alerts but tagging is one of the great one and so now I could set up a budget alert for the Money Team spend that I could say go over here I'm going to switch the tab I'm sharing um to our budget alerts feature and so now I'm an account admin I've set up some budgets of how much I

22:08want to spend I could add a new One for the Money Team um and say I want to make sure the Money Team doesn't spend more than $5,000 a month I'm actually going to set that off of the budget money team now I haven't actually used that so I don't think it's going to prepopulate for me here but but now I've set up a budget and I can say let's have that

22:31alert go to our team I can put our Alias at Databricks.com I'm not actually going to type our teamas in here um I don't want everyone emailing us and then uh we could go ahead and create that um I have one value specified let's see there we go and now we've got a budget set up so it's going

22:53to start pulling in data now I just created this one there's nothing on it eventually what we're going to be able to do is we're going to have two things uh we're going to have this burndown chart where we can see for what our team was configured to spend how our spending is going over time for the month and then when it crosses that it's going to email a team and let us know we've gone over budget and so we can set up various alerts and that'll make sure even if we get very busy with our work and we're

23:16not paying attention we can uh you know be notified hey and so one of the things you know a lot of applications that teams rely on for communication can let you create an email Alas and so you could create a slack room email Alias and then use that as what the budget alert goes to and then just go ahead and have your team you can also um do various Integrations where you set off

23:38like a pager Duty or something to your team based on those alerts I've seen a lot of people do things like that using those email aliases to basically trigger a workflow and have people log in and say okay what's going on why did my spend go over is it like this where we're just sort of spending a little more than expected every day or is it more like a case where there was a dramatic Spike one day and we need to pay attention to that um we periodically

24:01see things like that um where you know you have a dramatic Spike and spend we set up a fun one on our demo account to be able to show what that would look like I'm going to switch over to it let me get that to load up and so uh if we had a case like this where we had a single day instance as that happened we would have those email alerts go out and

24:23someone could log in and see okay what's going on this is unexpected let's shut some things down um and address the issue it's amazing so I think now customers can handle everything right to their cost you can literally uh put all the safeguards to make sure that you have a healthy consumption and you're not spending money for the sake of

24:45spending money by understanding the value of each project by assing tax okay this project cost costing me maybe $20,000 per month but this is the business value that's bringing from this specific use and you have all the tools available like other uh like built pre-built dashboards you can also build your own dashboards I've seen customers like doing some

25:09amazing dashboards like they were able to track like the tables that are not used and uh and with lineage understand the workflow that runs to update those tables and then maybe see if they can drop those workflows because may they are like just ghost workflows running without any value I've SE like and all kind of dashboard and they can just use

25:31system tables as you mentioned or and they can also rely on the on the existing one yeah absolutely at the end of the day the purpose of all this finop stuff is to make sure you're aligning your spending with the value you're getting from it companies are investing a lot of money in data and AI they're hiring teams they're getting all this

25:53data collecting it storing it they're setting up Databricks and other tools and the whole purpose of that is to drive some business value for their company um and the more that they understand that and put the rigor into it the more they'll be able to match those I've seen some customers go as far as using sort of a project tracker for each thing they do and making sure they

26:15have a tag created with that and understanding the value of that project like this is to make a decision that has this much impact on the business or this whole workspace runs an application that we collect this much revenue on and they can actually then tie those things back and so you see the most mature customers in this really understanding that value creating tags for the value using the

26:38same tags on the cost and then you can get to Roi um and so that's that's definitely the direction we're looking is how can we help customers do this now we know a lot about the cost information we don't yet know about all the value information um but as we help organize things by tag and as those tags correlate with an actual project we can have those conversations so we've been advising the Databricks

27:00account teams that work with our customers to say hey understand kind of the landscape what are the workloads what are kind of how do they break it down what is the value of that project now tag that project the same way and you can track it over time to see is it okay that that one's going up because the revenue they get from it is also going up or is that an area where they expected it to be more fixed cost and we should pay attention when it goes up and

27:23so I think those kind of conversations will get us all to a place of more maturity where really we can make sure that um everyone is spending money they're happy with because ultimately that's how we are successful no one's going to be happy if if Databricks customers are spending a lot of money um and they're not getting any value from it eventually they're going to shut down those workloads and so we want to match

27:45those two things um as closely as possible and tagging is a great way to start to connect the dots there all right so um we have the budget policies at the workspace level uh we can track the budget at the aun level build nice dashboards out of that you know to make sure we can track everything filter by tag by workspaces based on the organization what's what's next what's the next step yeah we see

28:09more and more companies where they have um you know workspace manager that's responsible for a particular team and today you can go ahead and create a cost dashboard and filter it down and share that with that individual um but it does take some work and you know if you only have two of them okay you just go set that up it takes a few minutes if you have a thousand different teams in

28:32workspace managers that starts to get more complex and so um one of the areas we're we're definitely looking into is how can we make that easier bring more of that cost information to that workpace admin and so um same information just sort of like at a different level of the organization more specific now anytime you do something like that you want to make sure you're

28:54thinking about security um privacy who is getting the information should have it and then that also depends on the company being set up in data Brooks in a way where we can match things to it so going back to something I shared earlier you kind of really need to get started and organize your resources in a way to make that possible um but that's one of the next things we're doing so it's really you can understand your Databricks costs how do you get down to the

29:16query level that's an area we've been investing how do you get the right person to see their specific costs and now you have an individual responsible for something that can see understand and manage that specific thing kind of at a much more fine grain place um where you can operate effectively than you want to learn more just make sure to watch Greg session at the data ni Summit in June and before I

29:41think we close I think we have something I thought about who's better than Databricks to monitor Databricks cost think this is something you need to meditate and think think about it anything else to add Quinton GRE no thanks for having me it's uh it's always fun talking about this um you know this is an area that customers care a lot about and we see a lot of energy going into it and I'm always trying to enable them to sort of

30:04like innovate and solve their business problems and then figuring out how I could scale that and take sort of the best examples I've seen from customers and make it really easy for every customer to then do it and so it's always exciting hearing what folks are trying to do um I love connecting um with with folks that are Hands-On with customers um so yeah keep feedback coming on these things uh we try to be

30:26very responsive here at datab bricks Asm product team and and just kind of always be very centered on what the customers are doing where their pain points are um and so looking forward to developing more great things in the future and I'll be back hopefully on your show sometime to share some cool updates amazing thank you so much Greg thanks Greg