Databricks Cost Management 2025: System Tables, Genie, & FinOps

0:04Hey folks, today I have the pleasure to host Sad Hanabala. She's a PM from the money team. She works in great team. >> Hey Ysef, great to meet you. Yes, I'm a product manager on the money team. My sole focus for the past year since I've joined Databricks has been everything around cost management. How do we help our customers understand, explain,

0:26observe, and track and monitor their cost? So really excited to share more about that with you today. >> Awesome. And [snorts] as you all know cost tracking and observability are a very important topics for Databricks and many companies and I think during the last couple months Databricks released so many features to help you understand your cost but also understand

0:48everything that happens inside the platform and I guess Sadana has many demos to show you today. >> Yeah, good luck to me. I have maybe five different demos to show you. So, let's see if it all goes smooth. >> Yeah, may gods of the demo be with you. >> But yeah, maybe to kick off, I would actually just love to recap thanks to our favorite app, Lovable here, uh cost management year in review. So, yeah, our

1:12team has been busy uh this calendar year. both the money team uh which is a team of really talented engineers who are building the best observability surfaces but you also have a very talented pool of essays like yourself Yousef who are very passionate about cost observability and helping us develop solutions and dashboards even genie spaces to make cost tracking easy. So year in review this year we've

1:35launched over 14 that that number is probably an underestimate system tables. Hopefully everyone's familiar with system tables but that is your one-stop analytical store. Databricks hosted tables that gives you observability into all the usage across your platform across all product types and user types and workspaces. Specifically the account prices system table is a private preview

1:58we launched this year. It's still in private preview. We're working hard to get that out to public preview. So for AWS and GCP accounts, you will actually have a system table that gives you all of your actual discounted prices per skew. So you can use that for more accurate cost tracking. We have four new private previews that have launched this year. And again, the team has been working hard and having customers try

2:20these features, give us the feedback helps us move fast and getting these features graduated. Everything from simplified native UI to make out of the box cost reporting super easy with zero clicks required from customer to more involved and sophisticated usage dashboard and even a genie space which I'll show today that allows you just to use natural language to understand your costs of the system. One I guess uh

2:44early Christmas gift that we just launched two weeks ago is a a new solution accelerator. One of the common questions we hear from customers all the time is Databricks costs are just a component of my TCO total cost to ownership. You know, I'm I'm running Databricks in my Azure environment or AWS environment. How can I co-analyze those costs? So, working with two very talented essays, we now have an

3:06accelerator with simple setup, you'll actually get dashboards that co-analyze your costs across your environment. And then one highlight of this year for our product team and uh myself and Greg attended for the first time the PHOPS conference. So this is a meaningful effort for us to get better in touch and build better relationships with the PHOps community, PHOPS practitioners. Um

3:28and actually this year we were we were contributors to the V1.3 schema of the focus standard. Um so really Databricks is trying to overall get you know um really meet the needs of our FIO practitioners. >> That's amazing. And I think it was very packed and yes, I've been playing with the new system tables. Um, just make sure to run the schema to have them

3:50activated for you and you're good to go. >> Exactly. And most of them are actually even auto enabled now. So that step should not automatically be needed. But if it sounds good to you, you can start some of the demos of these private previews. >> Yes, please. >> Okay, let's do that. So I think our brand new preview um that is and it's spanking new is the cost overview UI. So this is in private

4:12preview. If you're looking to get enabled, please reach out to your account team and uh we can have you enabled fairly quickly. What this is it's it's a native UI that's directly in the account console. Um and it gives you just very low low latency quick sense of the cost across your workspaces, across your products as well. Um, so this is a demo environment, so these

4:34costs don't mean anything, but you see I quickly get a sense of what are the biggest drivers in terms of product lines in my environment and I can understand the pace of growth of my cost as well. So this is a it's a new private preview. I think what is most exciting about this launch is not only is this available in the account, but for the first time we have built cost features directly into the workspace as well. So

4:56here you can see I'm in a demo workspace. I'm under the workspace admin setting in the familiar workspace UI and and here I actually get a scoped view of the costs of my workspace. So now workspace admins can um you know be more aware and and feel more involved with the costs of their environment. A lot of customers want to have workspace level cost transparency as well. So

5:19that's our our first product and again you'll see this develop with more capabilities as we build more granular filtering capabilities but the idea is to have native out of the box full transparency the cost of your account in your workspace as well. Yeah, that that's amazing because in the past you required the having access uh like having a meta store access privilege while now with just uh like with just

5:41the workspace admin privilege you can track the cost like natively without any um computer behind the scenes that everything is compute out of the box uh by Databricks. >> Exactly. Um but I know I know the only customers that here say you know Savana this this looks interesting but really we want more sophisticated granular uh filters. I want to filter by tag or I

6:04want to see my top 10 jobs. So for for those customers that want to have a even deeper dive of their cost. We release dashboards um you know periodically we update the dashboards and these dashboards will give you a personalized copy of an AIBI dashboard with all the pre pre-written SQL queries that you can customize and clone but if you come in here and you

6:26sort of visit dashboard you will be taken to this is version two of the usage dashboard so that's kind of the journey right cost UI gives you very quick simple snapshot of your cost if you want to get deeper you can visit a dashboard you can clone modify this as well V2 dashboard we actually demoed in the data and AI summit about you know five six months ago. Uh Cody Davis is you know very well-known SQL SME who

6:50helped build this dashboard. At a high level it'll give you some of the same breakdown to see cost by product, see cost by skew. We've actually built in AI forecasting capabilities. So this is based off the AI forecast function which is I think also in public preview. Um just to give >> AI exactly. Yeah. So underneath the code you'll see it's uh AI BI's AI forecast function is used. Um and then I think

7:14what is the best use of this dashboard is in particular for customers that have tagging framework in their account. You can sort of come in here. You can filter by tag. Here I'm looking at the owner tag. Again this is a it's a dummy environment and I am looking at the amount of cost that is matched to the owner tag. And I can actually even see here uh what are the top teams

7:38that are contributing cost in this account. So you can filter by tag. You can get a sense of the top tags in my workspace. Here you can see tests. I mean this is a demo environment. That's why the test key shows up so high. The budget tag, owner tag, endpoint source tag. And for customers, one of the biggest problems they have is like you can sort of see here, okay, we have

8:00maybe $150,000 tagged with this owner key, but really the environment is much bigger. What is what is untagged? Why is my tagging coverage so low? Uh we're building more and more granular capabilities. So if you go to the objects tab, you'll be able to come in here. Um I can filter for the owner tag key. And now I can actually see what are the specific assets in here that are not tagged or

8:23not matching the tag um owner. So in this case here, okay, I sort of see there's a SQL warehouse that's cost 150,000 in this time range. Um I can come in here and actually click in and if I uh click in, I will sort of see okay yeah that this is missing the owner tag. So ways just to build more operational capabilities into into this

8:45dashboard. Whether you want to view high level cost and forecast, you want to do chargebacks based off um different tags or you want to identify why your tagging coverage is low. The usage dashboard gives you far more advanced capabilities for understanding your system. >> Yeah, I love this dashboard. I mean, I've been using it for quite some time. Does help a lot understand from where the costs are coming. But and for those

9:07who are watching, make sure to tag all your assets, your SQL warehouse, your clusters, your uh endpoints, jobs, and if you're using serverless, you have also serverless budget policies that will automatically tag your your your workflows based on the group you're part of. >> Exactly. Yeah. I think this is the biggest takeaway. We've had customers realize by using this dashboard, what

9:30are the specific untagged resources and they have developed new tagging frameworks because of it. So you know and maybe okay the dashboard is great the UI is great but really what you want to do is you want to use uh an easy natural language experience to understand the cost of your naturally right we're living in a genai powered uh world now so again this is new and we

9:53built uh Newman Ali who a very talented essay in the DBSQL observability space and I co-built this genie space and at the end we can share links to all the resources so customers can download it. This Genie space is preconfigured. We've done a lot of fine-tuning against a benchmark and we've gotten over around 95% accuracy. Uh this Genie space will

10:16give you um a sense of your historical costs in your account. You can even do simple forecasting with the Genie space. And if you have the Genie research agent enabled, you actually uh there's actually even more advanced analysis that Genie tries to do to understand the top drivers or your cost. Maybe we can even demo and see some of that today and just show Oh, good. And just so you know

10:38how we built this uh we have taken advantage of all of the new capabilities Genie has built around >> uh semantic modeling and uh adding fine-tuning. So a lot of text information just to ensure Genie understands the scope of the data. For example, we don't have information on VM and virtual machine infraost. So we're

11:00telling Genie here, hey, if users are asking you questions about this type of data, please don't try to respond. We've given it example joins, how do you actually join the usage and the list price table? Uh we've given it common SQL expressions. So these are actually business concepts talking about semantic modeling and and metrics. You know, what does AI products mean? What are the specific products that are considered

11:23AI? We've fed Genie all these instructions given sample queries as well. So with a lot of fine tuning, we've built a pretty nice experience around um interacting with your Genie store. So let's see if we can actually um even ask a question to this genie right now. What is the highest cost warehouse this month? >> Yeah. And I really love like the new experience because it it explains like this like the that genies bringing like

11:47found relevant data in those tables and shows the different steps. So it can track what's actually happening behind the scenes. Yeah, it shows you all the traces and the thought pattern as well. So here you sort of see there's a a specific workspace this month in this dummy environment. We spel spent over 60k on it and then I can come in here see the code and based on my months of

12:09analyzing and writing SQL queries on system table I can tell you this is indeed a factually correct query. So that's good a genie wrote and uh let's ask something else you know how much am I going to spend next month uh here we'll see genie actually use the AI forecast AIBI function this goes according to plan >> and I think also the re the research uh mode will be even better because the

12:32research mode will will draft a plan with a simple question can you explain the trends of consumption for the last couple year it will it will build a plan and then run multiple queries and And uh at the end we're going to have her like going to have an entire report explaining all the different steps. I I've tried it in before and the results are just insane.

12:54>> Yeah, exactly. We could even try demoing that depending on how long it um takes. Okay, so here we have an estimated forecast. It looks like Jeie has given me daily forecast but looking at the code here I will sort of see. Yeah, you're you're seeing that it's using AI forecast. um as the uh as the logic for

13:16giving me sort of a a forecast. Yeah, here we are in December 20th and um you sort of see a I guess daily forecast of cost uh over the next month. So again, this Genie space is uh it's pretty brand new. We really encourage customers to use it. Uh the way you bring this into your environment is through the new import export APIs that

13:38Genie has launched. So uh the code is in GitHub. uh you'll be able to import this space and please give us your feedback on you know how this is meeting your needs. Okay. And then again the >> all the links >> in the description of the video if you want to deploy [clears throat] those. >> Yeah, that would be great. Um and then and the last thing that we're super excited to share is uh myself, Stephen Mushler, Daniel

14:02Martinez uh just released a new accelerator for TCO, total cost of ownership. So here I'm showing you an example of a dashboard that's actually pulling in both the Databricks costs as well as the Azure costs of my environment. So this is super neat. Um you know so here for example I'm sort of seeing across the last 60 days this we don't have a lot of usage in this

14:24environment. Um but I'm actually able to see all my Azure services you know that I'm paying for from storage to the actual azure data brick service cost in one place. Um, if I come in here, I can actually filter this cost by cluster warehouse. Um, in this environment, we only have one warehouse that's running. It's a serverless starter warehouse, but I can

14:46actually sort of see the Azure um kind of with serverless, right? So, the costs are all um in Azure Databricks. But if if I was running some non-serverless workloads here as well, I would see the VM cost as well as Azure cost in in one place. That's this is the last demo that we're um excited about. But we'll be able to break down your cost via tag.

15:08All these tags are actually propagated to the Azure cost management data set. This is an accelerator. We'll drop the link as well. Um, and if customers implement it, you'll get a one-stop shop view using AIBI of your Azure. And then we also have an AWS solution as well for the cost. >> That's amazing. I wasn't aware of this one. And I think it will it will also be a good addition to what we have because

15:30since we couldn't have like part of the private preview of account prices we couldn't have access to the um all the ris like all the cost related to the infra to the infra cost with this one we will be able to fill uh that that gap which is great. >> Exactly. Yeah. So based on customer feedback, we'd love to see how we can make this more of an inproduct

15:52experience potentially, but uh hopefully customers will find this one useful. >> And since this is uh let's say a year in review and you showed us so many things to release, so many features, what can we expect from or in 2026? >> Well, luckily we've done some thinking about this, what to expect in in 2026. So I think there there are three areas

16:15of interest and we would love to hear feedback on if we're going in the right direction. One is you know the focus of 2025 has been okay system tables low latency high granularity data how can we help customers build more visualizations and reporting experience around it. In 2026 where we're interested in is bringing more intelligence to that experience. So more agentic workflows,

16:38more sort of opinionated insights on your um sort of observability and and Databricks environment. So that'll be one area just AI powered intelligence. The second one is we showed this in one of the demos but you know we understand that account scoped cost transparency is not enough right? There are a lot of users that want cost transparency at the workspace level maybe even at the resource level. So

17:01building more of a federated cost transparency model. So this information is present at the workspace level as well is very important to us and it'll be an area we're interested in in 2026. I think lastly for a lot of for serverless and some of our product lines we have very high cost granularity you can get granularity down to the workload level down to the user level for other types of compute there's still areas where we can give further granularity

17:23and breakdown. So one example is SQL and uh Juo and his team they launched a private preview this year on how we can actually get SQL costs not at the warehouse level but down to the query level. So using that sort of framework we're interested in in giving more and more granularity and drill down capabilities on areas where costs are shared across a compute container how to actually break that down whether it's clusters and warehouses. It's an area of

17:46interest that we're looking to build more. >> That's that's amazing. I'm really looking forward to what you you and your team are going to be releasing. And for those who are watching the record, the recording, make sure to leave a comment to Sadana in case you have any suggestions or maybe reach out to her directly in LinkedIn. Thank you so much, Sadana. It was a real pleasure uh to

18:10host you and I'm pretty sure you you going to be my last guest for 2025. saving costumes everybody. Last but not least is uh >> Good Moon, Yousef. But no, thanks for having me. We're looking forward to hopefully in a year from now giving you um a lot more of these goodies. So, >> perfect. Thank you. Bye. game. Hey