Data governance with Unity Catalog on Databricks

0:04I have the pleasure to host two bricksters Karthik and Kiran. They are both behind the first Unity catalog book called data governance with unity catalog on datab bricks on in partnership with O'Reilly. Hi. >> Hey. >> Hey. I work as a specialist solution architect with datab bricks. I come with a decade long background in data

0:27engineering uh starting with the Hadoop definitive guide in 2012 like you would probably all have read that book who has worked on the big data platforms and those days right after working on the data engineering platform so for long I joined Databricks back in 2022 and spend a lot of my time working with customers on unity catalog and especially data governance so I've been

0:49working on data engineering plus data governance use cases for a lot of customers of Databricks in the in the IMIA region and today I'm going to talk to you about our book and a bit about data governance in general using unity catalog. Off to you Karthik. >> Thanks K. Hey everyone, I'm Karthik Subaro. I'm also in the same specialist

1:11solutions architect team and I also focus on platform security and governance topics here at databicks. My background is basically application development, architecture and everything related to big data, data engineering and stuff like that. As well as like I had some experience working on the machine learning parts of things. The time when AI was not that popular now it is uh and now I also focus on the end

1:33toend architecture for data and AI uh landscape within uh several organizations. I work with many customers here in the central European region. >> That's awesome. Let's go straight to the point. What is the reason for writing the book? >> Okay, I I'll give it a start. So, a couple of years ago, we thought about writing a book because that's something we wanted. We both me and Karthik both

1:57wanted to be an author and what a good platform to be on than O'Reilly, right? Publishing a book with O'Reilly. So what happened was that at that time we both were working with a lot of customers on unity catalog spending a lot of time with customers implementing UC and the data governance side of it. So then we thought okay we have a lot of best practices uh which we have learned from

2:20our engagements with customers why not just publish it as a a document that way of in the form of a book right with O'Reilly. So we reached out to O'Reilly and we put a proposal for publishing this book and it got approved and to be fair like we spend a lot of time analyzing okay what can be the content for the book. Do we have enough content to be published as a book for unity

2:42catalog but when we actually started researching we found that unity catalog is not just access controls and data governance like it it expands to a lot of other parts of the product itself. And when we started to write our content for the book, right, we actually had to eliminate a lot of things which because we couldn't fit it into a 300page book with O'Reilly. So that's

3:06how it all started and it took us more than an year to like actually write the book and get all the reviews done and we saw an opportunity in the market. There was nothing very comprehensive about Unity catalog in the market and there was a place for us uh to publish a book and we be the first person to come up with a very comprehensive best practices. The approach we took when

3:29writing the book itself is right we took a sample uh company a fictional company who started with HMS and then started using Unity catalog for their data governance. how they grew from a very early startup to an enterprise company and how Unity catalog was a catalyst in their in their growth. So that's the approach we took in the book and I hope like you all would like reading your

3:52book as well. >> Exactly. So just to add to what Kiran said. So something that we had been talking to customers a lot. We have been suggesting and recommending certain best practices and we have seen a lot of content like we have a well doumented uh public docu documentation from Databricks as well as there are several blogs but the content was like here and there as well as like there was no story

4:14for this end to end approach and that's why I also kind of wanted to put together a book which would tell the story from the beginning until the end with an example of a fictional company that would go through this journey itself and then adapt and expand and their database footprint as well as Unity catalog governance footprint within their organization. >> And according to you, how we can how can we define Unity catalog for beginners

4:38and what was the story or how did the uh catalog story evolved uh within Databricks? >> Exactly. So, Unity catalog is what we term like within the datab bricks u platform we term it as the governance solution. So if you look at the story of datab bricks itself. So it evolved into what we now call the lakehouse platform.

5:00Initially it was like decades long of two different systems like one is the data warehouse and the one is the data lake. So one dealing with the structured data set one dealing with unstructured data set. So datab bricks evolved into a platform which would support both of these use cases like all the unstructured data set and structured data set in one place. you would build all kinds of use cases, BI use cases, AI

5:22use cases and stuff like that. So this unification also required kind of a additional security and governance which simplified end to end use case implementation and this is what unity catalog brings in. So it's a governance solution and it also has a lot of capabilities that is necessary for anyone to implement like any kind of use case for that matter in terms of data

5:44and AI. And that's why unity catalog is important because it kind of enforces security and governance by design. So here the security and governance is not an afterthought. It is by default when you start from scratch. This embedded into the processes. It is embedded into the implementation framework. So that means you don't have to worry when you go to the production. So most of the use cases that we see in the industry is

6:06like everything goes well in P and also maybe to a certain extent in MVP. But when it comes to the real production deployment, things get a bit difficult because you have to kind of incorporate all the secret and governance mechanisms. But with Unity catalog, you kind of start with that from scratch. It is easier, it is simpler and it is kind of brings in the unified and open approach to governance across all the

6:28use cases. >> And I know this is something that was a little bit confusing first. I remember we had many type of compute. I think first it was called single user and if I'm not wrong shared now we did rebrand them single user is now called dedicated the shared called the standard mode this was a little bit confusing so according to you how should the users choose the

6:52right UC compute type for their workloads >> yeah that's a that's an interesting question so a lot of customers are still confused like what's the compute I'm going to you choose for my workload so before going to that uh just to talk about the evolution part of it a bit more. It's in 2015 that the data platform was integrated with highest. Now it's like 10 years. We are in 2025.

7:15We have Unity Catalog which was first introduced in 2021 and in 2024 Unity catalog became open source. Now when we were actually starting writing this book it was a proprietary tool within the data platform and during the course of writing this book we announced it as an open-source tool. So you can imagine like how our writing style would have changed. We may have to we added a new

7:38chapter itself for open source unity catalog during the course of writing this book. So that that's the kind of evolution that the product had gone through from 2015 when we first introduced HMS to when where we are today with Unity catalog. Now coming back to the question itself see we have multiple compute access modes in in Unity catalog right the first one the

8:00most common one is the standard access mode. The standard access mode is the default compute mode for any user today except for some specific use cases like distributed ML. But the good news is that we already announced a public preview for distributed spark ML workloads in a standard access mode. So you can now actually run uh SparkML workloads on a standard access mode

8:23which is kind of a share access where multiple users can come and run their workloads in the same compute. Now there is an architectural difference between the standard access mode and the next access mode we have which is the dedicated access mode. The dedicated access mode is using the original spark architecture right where you get the complete access to the compute and

8:45the storage within the compute itself. So it's kind of an elevated permission the users get when they run their workloads in in the dedicated access mode. That's why when you are running an ML workload, you have to only use the dedicated access mode. Now when it comes to the standard access mode, it uses the latest technology which is the client server kind of architecture

9:09introduced by Spark Connect. And using this technology, you can only run workloads that support the data frame API. So if you look at the standard access mode, we do not support anything that is RDD related or the old way of accessing files from DBFS. So all of that is unsupported in the standard access mode and only supported in the

9:31dedicated access mode. Now that has also created confusion. Okay, if I have to run an ML workload, I have to create compute for all of my users, right? I have to create a dedicated access mode compute for all of my users within the organization to run ML workloads. And that's the reason we came up with the new computer access mode which is a group cluster assigned mode where it's

9:53simply a dedicated access mode but you can assign it a group. So you you have a group of ML users. So they are all part of a group called ML users which which can be an intra group within your Microsoft intra and you can assign this dedicated compute to that group of users. So everyone can share the same the same access across this group access

10:17compute itself. Now there are multiple versions of this there there are there are positives and negatives to each of these these compute access modes in unity catalog. But if you want to read more about this you can read chapter 4 in our book and that talks specifically about compute modes and compute in unity catalog. Back to you. I mean that was

10:39pretty good summary of like which workloads or which type of compute should I choose based on of course the workloads I'm going to run and I know that access controls are part of uh the UC governance but how good is the UC access control features? Yeah. So the access controls are the core part of any governance solution including Unity catalog and the way Unity catalog

11:03handles access permissions is basically the approach where we took is the access control lists where you can assign permissions to certain users groups or service principles on specific objects within the Unity catalog. So it can be any kind of tables, views or it can also be a machine learning model or it can be a function, it can be unstructured set of files in the form of volumes. So it

11:25is a unified approach to kind of providing access controls to all kinds of data and AI assets. On top of that, we have also introduced this attribute-based access controls which is currently in public preview. So we also covered this a bit in the book and uh this approach enables organizations to kind of implement complex or advanced governance requirements like defining a

11:47central policy then enforcing those policy in addition to having access controls per users and groups etc. And this is also being enhanced by some adjacent access control features such as like workspace catalog binding where you can define who can access a specific catalog from which workspace for example as well as some additional uh controls. So there is also like um additional

12:10features being introduced into the attribute based access control as well as there is an upcoming feature which is also was introduced in DAS last year. So that explained how you can leverage the new role based access controls with exclusive groups. So that is also something that is upcoming. So the point is with Unity catalog you have a lot of um flexibility in terms of controlling

12:34access to your data and AI assets and this is also being enhanced and improved and you can leverage more and more features as we go ahead. So most of the core access control concepts have been covered in the book and uh you can go and dig deeper into that. >> We all know that sharing data is critical. Data engineers are going to prepare the table for us, compute KPIs, but how will UC help me share this data

12:59or not only data but data AI models and much more in a very secure way? >> Very good questions. So this is something that we see very often with our customers like when we speak to many organization big or small. So they have requirements that they have an asset that is they know that that is useful and they want to share it with either within the organization like to a different business unit or a different

13:23legal entity within the larger umbrella of organization or it can be with externals like for example take a retail example. You can share it with your suppliers. You can share it with your customers and things like that. And this asset sharing requirement is not just restricted to tables or any kind of structured data set. It is also relevant for sharing a really cool machine machine learning model that you built that you think would be useful for some

13:46other business units for example and this can be not just within the scope of your region or geographical location. So it can be that the one who wants to consume your data or AI product is sitting across the continent. They can be in other cloud region or they can be using a different cloud provider for example. So you might be on Asure and your uh recipient can be in AWS for

14:11example. So in these kind of scenarios it becomes tricky and also sometimes expensive uh to kind of share the data and AI assets or data and AI products. With Unity catalogs and adjacent features so to say like the data sharing which is a managed data sharing which comes as a part of the solution within the datab platform. This becomes much much easier because data sharing provides a secure and most efficient way

14:35to share data across regions and clouds and this is something that is can be leveraged even though the recipient is not using Databricks. So because data sharing provides Databricks to Databricks sharing as well as Databricks to open sharing. So this kind of setup provides a lot of flexibility security as well as options to securely share your assets. Why is it related to unity catalog? So data sharing is also an open

14:57source protocol which is independent of unity catalog but within the database platform you get both unity catalog along with data sharing. So when you start using Unity catalog so it kind of kind of a managed solution within the database platform and it's very powerful >> and we know now with the rise of uh the genai we have more and more a IML use

15:19cases so how do we handle those use cases with unity catalog so does you see also govern AI assets >> yeah that's a very very good question so basically why we say that you should run your AI workloads on Databricks is because we don't want you to take your data to your model, right? Most of the models out there, right? When you are

15:41working with those models, you need to bring your data to the model. But at Databricks, we took a different approach. We bring your model to your data within your security parameter. So all your data is already governed, access controlled using unity catalog and the model is also available within the security parameter of Databricks. So in that way you can bring your model

16:03to your data and make sure that you are making secure decisions using AI using your data. Now when it comes to Unity catalog how it integrates with AI models. We can register your your custom models in Unity catalog. We also bring some standard models as part of our offering itself. Now these models need features right. We also can support

16:25these feature tables in Unity catalog. You can discover it using your our discovery functionality and the most important thing for agent when you're building agents you need tools mostly functions right and these functions can also be part of unity catalog and moreover if you are interacting with agents like genie in in Databricks and

16:47you want to have some kind of metric capabilities so we also support capabilities like defining metrics in unity catalog that can also be consumed by your tools. So you you get the full control of all your data and assets within Unity catalog. And the next best thing about this is that we integrate with something called as our our AI

17:10gateway. So AI gateway persists the governance telemetry in Unity catalog including the auditing and and also who is actually using these models from from Unity catalog. All of that information is audited and available within Unity catalog in our system tables as well. So the integration between AI gateway that defines most of these guard rails for AI

17:34usage and Unity catalog enables you to have a fully controlled environment where you can use AI models for your use cases. >> Yeah. And we have a dedicated chapter for governing AI in our book. >> Yes. Look at chapter six if you want to learn more about governing AI in Unity catalog. >> And what are the steps I need to follow or like the resources to make sure that

17:56I master Unity catalog and can I use also the uh Databricks free edition to do so? >> Absolutely. So that is something that is getting more and more easier like as the unity catalog was introduced and now being adopted by thousands and thousands of customers and from the documentation perspective from the resources perspective it's also kind of exploding. So you have very well structured clean

18:20documentation from database like the public documentation itself is a great resource you can start there for enhancing your knowledge further you can also explore databicks blog as well as like of course like YouTube channels like yours yourself so that's also a great resource for learning like the next gen do watch any videos related to catalog that's also really helpful as well as you can explore for example we um also have a medium publication on

18:44which is related to like data governance so it's basically the article It's also written by SMMES like Iran and myself and this is also a really good resource you can explore that on the medium website as well as like you can go and read the book uh and book you can also get it on the Orley platform itself if you have a subscription with or you can just uh grab a copy from uh Amazon or you can also have like for example um go

19:07and go to unity catalog.io IO and explore the open source version of Unity catalog. You can set it up on your on-prim system or any kind of cloud engine then just explore how a catalog can function and how it interacts with Delta Lake, how it interacts with Apache iceberg and things like that. And uh you can also explore and practice with your with the database free edition. So that is also something that you should try

19:29because that is something that has been really wellbuilt. It is something that is refreshed because we also had a community edition before but now we have a much much better version with which includes like governance solutions like Unity catalog built in so you can explore there and uh learn for yourself. >> I think that's that's amazing. U you're going to find all the links that Kartik mentioned in the description of the

19:53video. I think that's it. Thank you so much Kiran and Kartik for the