Fine-grained Access Control at Scale with Unity Catalog

0:04Hey, today I have the pleasure to host Deepankar. Hi Deepankar. Hey you soup. How are you doing? I'm doing good, thank you. Uh can you introduce yourself? Yeah sure. Uh so I am with datab bricks for close to three years now. I'm a specialist solutions architect in datab bricks and I help customer build their

0:26solutions uh especially focusing on the data governance piece on our data intelligent platform in datab bricks and today I think we have the pleasure to host you because you're going to be talking and showing us an example of the famous and awaited feature attribute based access control in Unity catalog. Yeah. Yeah. That's right. That's right.

0:49And um uh we are very excited about this feature. Uh this feature has been requested by many of our customers. And uh today what we are going to talk about is doing the fine grand access control in what is possible in the way that uh is generally available and then we'll talk about attribute based access

1:11control. But before we start talking about all those, we also need to kind of understand there is something called govern tag or tag policies which is kind of centralized way uh in UI, SQL or REST API uh in managing your tags which then can be used to kind of implement the attribute based access control and we

1:33will look at those uh during our demo. Um now the other thing that is very important to understand is why ABback right because we could literally in in Databricks today you could uh define an UDF or userdefined functions and then apply that UDF to your table uh to kind of uh get that fine grain access control

1:56working for you. So you could say that okay I have a table which has data for US region as well as European region and I want my users from the US region to only look at data from US region and then you could also say that I want my non-admin or non-privileged user to only look at certain data elements but not look at sensitive data elements let's

2:19say such as SSN or data path or salary if it is uh something related to an employee data for example so you could achieve that using the UDF like I was saying and then applying those UDFs altering the table to apply those to the table but then you are going table by table so that's not really scalable in a lake house where your users could have thousands and thousands of tables so

2:43what really we are introducing with a is that scalability of applying this fine grand access control on your catalog in your schema and across your uh tables. So you basically define a tag and then you define who has access to those tags via tag policies and then you apply

3:05those tags to the table and there are automated ways to also get those tags applied and then you define certain policies on those tags. So now imagine if you define that policy at the catalog or at the schema level any tables within that catalog or schema which has columns with those tags can now be controlled by that policy. So that's that's the

3:28magnitude or the proportion or the scalability we are talking about here. And um you talked about also govern tag. So I believe you will show you will be showing us a demo about um I think we can move to the demo. I think it will be much easier for to understand how they Yeah. Yeah. Absolutely. So, let me go ahead and share my screen and so so this is our topic

3:52today. So, let me move to our environment where we are going to go to the catalog explorer. So, this is the catalog explorer where you could uh basically discover your data assets that are governed within your lakehouse. And then you will see something called tag policies. By the way, I must mention that this is currently as of the recording today, which is 25th April

4:142025. This is still in private preview. So, you need your contact your databicks counterpart or your um SAS or AES to get this enabled in your uh databris account. So, this is how you access the uh govern tag or the tag policies. And you can see on the right hand side, you could create your own tag. A tag could have u any number of values starting

4:38from zero. So you could create a tag just without any value and we'll see an example of that and then you could put a description for the tag and then you could put some allowed values. Now also you see at the tag policy screen you see account permissions right? So the account permissions is who has permissions on at the account level to do certain things

5:02in in the in the tag, right? Um so I could select a principle like I could select you. Um and then I could say that you have three levels of permission either create, manage or assign. So and it kind of gives you those description right. So a user or a principal could create a tag or it could manage a tag

5:24which is already created or it can assign the tag to some of the resources such as tables or columns. So basically you can create a policy call it for example PII then allow the value for example security number uh birth date uh I don't know uh country and then you can just give me the permission to add those

5:46tags to a specific table. That's exactly right. And then here is an example. So I created a tag called geo region. I don't have anything here as in terms of value but I could add a new description here and then I could give you permission. So I'm the creator here. So I have manage but I could assign other principles to and then I can say that that principle

6:09has assigned permission. So then in turn they can actually use this tag to attach to one of the resources. Um so this is actually at the tag level. The earlier permission we were talking about that was at the account level. This permission for each tag also you can have granular permission like I showed you. And then um so this is an example of a tag where you are just identifying

6:32the key not any value. But then you what you are talking about let me um show you an example of that that I have used for my demo is sensitive PII. So when it when we talk about PII there are multiple types of PII that we could identify and it could also change depending on the context or the country or the region right so in this case for

6:54my example I said that there are two um PII that I'm dealing with here one is SSN and the other one is address so then we'll see some examples where we apply this PII to the table and then apply the policy on the PII tax and that in turn mask certain data Um before we move to the demo, do you

7:16have any other question you? No, I think it's all clear. We can set the um I understand that we can set the permission at the account level but also at the tag level which is which is cool because you can just give some specific tags that you want to assign to a team so that this team can tag the tables that are relevant. So we know that if they're using this tag then this function can do hashing or encrypting or

7:39whatever we you will show us later. Exactly. So now let's actually move to a demo with some data examples. Um so in this case I have used this notebook where I have uh created some synthetic data. So as you can see I have purposefully created some data elements with first name, last name, date of birth, age, gender, address, SS and so

8:01on and so forth. And this is a sample um I just queried that data. Nothing is implemented yet. So this is a table with the PII data and it has uh all these data elements in it. And then I do some more um you know update and deletes and inserts um to uh create uh some more uh

8:24records and then I also create another uh table. The reason for creating another table by just deep cloning from uh the other table is I wanted to show the scalability because the example here I will show that I am applying the policy at the schema level but then I don't need to do anything for each table individually which I was talking about earlier but it will automatically by the

8:48virtue of having that tag associated at the column level it's automatically doing those activities or applying those policies that it is supposed to uh and then here is some um summary information. So I have data for all these different region. Uh in this example we will see some uh data elements specifically focused on the

9:10region. So that's why I'm showing there are certain um different regions available and now the actual activity begins. Now first thing I'll say that there are ways to automatically identify data elements and then put the tagging. So that's like autocl classification and autotagging. In this case we are just using SQL to uh apply the tags. Remember

9:34earlier we have created these tag in the tag policy. And remember for the sensitive PII I had two values SSN and address. So now I am just attaching those tags in the in the particular column of this table and actually both the tables and I'm identifying those

9:56sensitive PII SSN for the SSN column. So let's actually look at this how it looks like here. So if you go to the catalog explorer and click on that table you will see that these tags are now showing up here in the table columns. So now going back H just for the viewers keep in mind that uh Deepankar is showing you how to do this with the uh with SQL but

10:19you can also do them through the UI right right here you could um yeah you could just select this and you could do that and you can do save I'm not going to save it but that's how you could do it easily with the UI as well okay so now I have all these tags similarly I have associated or added the tag called address with sensitive PII now there are

10:42two things right we talked about one is a tag key which has certain values and then the tag key which doesn't really have any values so when you have values the way you would uh set it up is like this because I'm here assigning the tag to a column called address versus here I'm assigning the tag to a column called SSN whereas go region if you remember from our um earlier section of this talk

11:08go region really doesn't have any value so we are just putting it tax to your region. You don't need to put anything like that because uh that doesn't exist. So that's the first step. You define the attack. Either you have an automated way to identify those tags for the columns or you um add yourself. And then you define a function. So in this function

11:30I'm creating a very generic function. So you could uh literally provide any column and column value. I'm just saying that certain column are not when when certain column values are not equal to certain values then I wouldn't allow those rows to be returned right um and then I am this is the way you would drop

11:52a policy if the policy exist and then this is actually where the actual magic happens which is you are defining the policy so let me walk you through you are defining the policy with the policy name and then you are defining it at the schema level. So this is where the inheritance come into picture. Anything that is defined at the schema level now is applied whenever the tag matches

12:15certain condition matches to all the assets within that schema. So if I go up here and in the catalog explorer and then go up to the schema after I created that policy that also shows up here, right? So we have this hide west region customers that you can see it's also showing up in the UI itself. And in fact

12:37you if you want you can in fact create a policy from right here in the UI as well. Again we did it through SQL but you can do it via UI as well. Okay. So let's go back here in the notebook. Then I could put a comment and then I am doing a row filter using this policy and then I'm saying that who is affected or

12:59who who to whom this policy is applicable. Right? So this is a group or account group which is where this policy will be effective. So any user who belongs to the group this policy is for them and this policy is for tables where the column has a match which has a geo region. So earlier if you remember we have a particular column where we said

13:23that the tag is geo region. So for both the tables there is a region column and we said that has a tag geo region. So with that I am saying that if there is a tag for any of the tables which has a matching column that has a tag of geo region which I am aliasing as region this this line is mandatory by the way. So if you have to use this uh line

13:48number 10 then you have to have line number nine. So so basically saying the geo region and aliasing with region and then I'm using the column. So this is where the actual um UDF which we defined earlier is getting invoked. So where the region is west. So now let's look back how this is actually going to work here. So the column one is region and then the

14:12value of the column one like I was saying is passed as west. So then when whenever the value of the column is not equal to west show me those right. So any user who is part of this demo fine grand access control group one when quiring a table which has a matching column which has a tag of geo region and

14:34if there is data for west it will not show up. So that that's why the comment is hide uh customers from the west region for sensitive tables. Okay. Um so that that's that's basically it and then the function name is given here. So if you see this is the function name. Okay. So let's look at the effect now. Um any

14:56any question before I show no all of crystal clear. Okay. So again I had two groups. I'm just showing here that the user belongs to group one to whom the policy is actually applied. In this case the user is just me. I I ran this and if you are running this notebook we can share this. I think there is definitely a way to share this notebook. So if you are running this it just shows your

15:18group membership uh because it's actually evaluated during the runtime. So in this case now that that tag policy is applicable at the schema level. So the schema where this table belongs has that policy and I'm purposefully trying to query west. I am being part of that group. I'm not allowed because not equal

15:40to I'm not allowed to look at any row from west. And interestingly like I said you would not need to individually go and apply this to the tables because you applied it at the schema level. So all the tables are showing the same behavior. So delta 1 2 and delta 2 sorry delta 1 and delta 2 both has now uh suppressed the records for west region

16:04for this user who belongs to that group one. Um and then here I I'm just doing a select star and you could see other regions are coming up but not west. I could query midwest but not really west. Very similar way uh how the row filter works with aback the column masking works in the very similar pattern. That

16:27means you define an UDF and then you apply those as part of the policy. So in this case I'm saying that any column that matches with this column one it will just be returned as um this x right. So basically masking and then I am defining two policies here one is again the construct is pretty much the same it's name of

16:50the name of the policy and then where it is applied. So in this case on schema and then uh using the evac column mask which is the name of this uh udf here and it is again applied to the group one but in this case we have the flexibility of saying that certain groups are excluded. So if a user which belongs to

17:14that group uh queries then they will still be able to see the actual data. So this is where that admin versus non-admin or privilege versus non-privilege type of implementation comes into picture right because if certain admin group has the ability to see and if a user who is part of that admin group now queries the same table they are able to

17:36see the SSN for example again it's matching that columns it's pretty much the same and in this case we are using hash tag value so I want to mention that because remember region doesn't have a value so there you have to use hashtag function versus in this case you are using hashtag value and then you say that sensitive PII remember this is a key value pair so you have to give the

17:59value and again that aliasing this line is mandatory and then on column SSN right so basically use this function to apply the policy to this group on any tables which belongs to this schema and uh which has a tag of SS that's how I'll read it very similar uh condition for address. So now again uh this user which

18:24is myself in this case is part of demo of fine grand access control group one where the actual policy is applied but it's not applied to group two right because we said except group two um so immediately we could see uh remember this same table I am using so there would not be any west data plus since I marked address and SSN I'll not be able

18:46to see those two and uh same with the delta 2 again the scalability so the um the policy is applied to any table within that particular uh schema where the policy is applied to um so I can quickly go here so you can see mask address and mask assistant is created and again west region is nonac

19:10uh this user being part of um group one now I'm just uh making it little more uh complicated a little more fun I added the same user to both the groups. Now when there is a situation where uh you have certain group and then you say accept certain group then the precedence is given to the accept part. So the user

19:33is now able to see um the SSN address. So let me clarify that you remember this part for example you can say all the data engineers won't be able to see this except for example the manager of this team. Exactly. Exactly. So in this case see the same user belongs to both of these groups. But um since I have this

19:57except text precedence. So when I um when I changed the group membership here you could see I'm part of both the groups. Immediately these two are unmasked. Um same is true for the other table delta 2. uh but of course west is still blocked because the row filter policy didn't have anything with except

20:21and then when we add the except I think it's kind of becoming repetitive but you you know what I'm trying to show here that the west shows up because the user is part of that um both both the groups the other thing I wanted to talk about is versioning so uh when you apply u the the policy on a table and then if

20:43you are not in the exception group now you understand the concept of exception then you would not be able to do time travel. The table allows you to do time travel. Example of this is uh and remember initially I said that I did some additional work on the table after I created. So I did two more writes and it created three versions. The initial version and then more two more. Um, so I

21:06could query something like this when the uh when the uh when I'm trying to access the version one in this case and I'm able to do that because I'm in the accept group. Now if I remove the accept clause so you you see that I have changed the policy because you could literally do this create or replace um

21:28and you see that after these two I am directly going into the table. I don't have the exception clause. So immediately I get this that I cannot do the time travel. So this is very important to understand. If you want certain group of users to be able to do time travel then put them in the exception group. And I think uh that probably it. I just added few more things here just little more complicated

21:51scenario. So you could actually define your UDF or functions with more uh parameters. So the signature is little different in this case. So like I am saying complex. So in this case you are saying that okay if certain values are matching then um provide those values otherwise mask it. So I was just trying to do some fun experiments here. So if

22:15the gender is then show the SSN otherwise mask it um and then the way to do that is if you want to uh provide additional um parameter values for additional parameter the way you will do that is using. So that's important because this is an example of that where you could actually keep adding more parameters. Um yeah I think that is it

22:39YouTube if you have any questions we can take that now. That was that was all good. Um thank you. Thank you so much for this uh getting started video. So for those who are watching you can see how easily you can define your aback uh using the UI SQL as well. You can do column masking, row, row masking, row

23:01level masking, filtering. You can define group exceptions. You can grant permissions over tags at the uh account level, at the tag level. And of course, you have the inheritance. So we don't need to apply the masking at every table whenever we have tags, then this rule will be automatically applied. Thank you. Thank you so much. Thank you so

23:23much, Japankar. And see you soon. Yeah, see you soon. Thanks for the opportunity. Thanks for having me for