Deploy Unity Catalog with Terraform on Azure

0:00huh hello and welcome to this new session uh today I have the pleasure uh to have uh himanshu with me so uh imashu will show us today how we can deploy Unity catalog in Azure uh databricks environment this video is a follow-up regarding the

0:23previous video that has been made with Yasin regarding how to use terraform data breaks examples and I leave this the stage to himachu thank you sir thanks for inviting me for this session so as you have introduced I'm gonna show you how you can deploy Unity catalog using terraform on Azure

0:45databricks so this is the GitHub repo which contains all the examples I think you have already seen this in the previous video so I'll focus on one of the examples in this uh repo so this one ADB Unity catalog basic demo so basically this one deploys uh JT catalog end to end on Azure recovery so let's dig into this

1:08so let me guide you through the um the whole process what we are going to do in this example and then I'll uh also uh go a bit deeper into the code so first thing that we need to do is deploy a meta store inside UT catalog metastore is the let's say the high level uh container envelope that holds

1:30everything so you can have like one meta store per Cloud region to deploy a meta store you need a storage location where you know uh all the metadata about your manage tables will be stored and you also need a way to connect to that storage location like in case of azure uh we are going to use manage identity which will allow us to access ADLs account uh storage account from beauty

1:52catalog service so basically to create a meta store we need these two things one so basically we are going to create first of all the manage identity then we will create a storage account we will allow the manage identity identity to read and write into the storage account once we have these two objects ready then we will create a meta store in one of the cloud regions then next step uh would be once the meta

2:15story is created we can obviously see that available in the databricks account console uh then we um but as a Next Step uh we uh need to attach uh one of the workspaces to this metastore so This Is How We enable a Unity catalog into one one or many workspaces so again this we can do on

2:38the account console but in our example we are going to do this using terraform once we have done this step then we are going to move to the step five where we are gonna sync uh some of the users and groups from Azure active directory to database account using terraform so basically you can do this part either using steam or in this example we are going to do it fully automatically using

3:02terraform so to do this to do this part we obviously have to create the groups the concern groups beforehand in the active directory then only we can obviously run this template uh to bring them to this account after bringing those users and groups to the account then we can also kind of sync those groups and users from account

3:24to the workspace uh using a feature called identity Federation again we are going to do this using terraform so this is more like a setup on the account level now we are going to also create some UT catalog objects on workspace level so we want to let's say in this example we are going to create a catalog for Dev environment we are going to call this Dev catalog again to create

3:47a catalog we also need a storage location that will be used as the managed location for this catalog where all the manage tables will be stored so for that we are going to create an external location inside Unity catalog and to create an external location we obviously need a container or a location somewhere in ADLs and same way we need storage credential which is obviously

4:10going to use a service principle or manage identity that will allow us to access this this location so once we have created this external location then we can create the catalog uh who will be using which will be using this location as a manage root storage for this catalog and once we do that part then we can obviously create schemas or databases for example in in our case uh

4:32we are going to create a bronze silver and gold schema and also we are going to give some access rights and permissions to these catalogs and catalog and schemas to different groups and users that that will be available in the workspace so that's the whole uh basically end to end setup that we are going to deploy using Telecom all good yourself any questions all clear cool

4:55then let's uh dig deeper and we are going to see some of the terraform template for these objects so uh as I as I showed you earlier basically you can access this example from here uh and then we are going to land up in this page so let me start with the main.tf file where we are obviously going to start by

5:19you know let me let me zoom in a bit Yeah so basically here we are starting you know uh by adding the providers like Azure resource manager database provider uh we also are creating some local variables or local local variables which obviously we are getting from uh different variables so let me start by the variables I think this is important basically we need three variables to run

5:42this example first one that we need is a database resource ID so it's basically a resource ID of one of your database workspaces that already exist and you need that to Because unless you have one workspace uh you can't access the account console anyway so you need to create one database workspace on Azure and you can get that resource ID from from the Azure portal I can probably

6:04show you uh like I have this I have this workspace which is already available and in the top right section you have this Json view which gives you this resource ID so basically we need to copy this this gives you the subscription ID the resource Group where you want to you know deploy your resources we also need the database account ID and

6:26the list of active direct Azure active directory groups that you want to sync to the database account right so in my example I have already created couple of groups in my active directory as you can see uh there's a group called account Unity admin which is going to be admin of everything inside Unity catalog as well as account admin I've also created three other groups like data analysts

6:49data engineers and data scientists I also kind of added some users to these groups for example in the data analyst group I have a dummy user user one I have a search principle if I go back to the admin group so basically I have added myself as admin of anything inside Unity catalog as well as the account database account so basically we need these three variables to run this

7:11template uh in this terraform.tfr files so basically you can add your variable values here if you want before running a template uh so I've like left some placeholders but obviously you can put your values before running the template for the user groups since I already knew what groups I need to sync so I've already added those groups right away in this file okay so let me come back to the main uh

7:35dot TF file so I'm just creating some local variables from this bypassing these variables then I'm you know uh initializing the Azure resource manager uh uh terraform provider with the subscription ID that I provided from the from the variables and yeah I'm also getting some more information about my workspace like workspace URL and all

7:59those things that I will be needing uh in this in this template then I'm also uh so basically one one important Point here I'm initializing data frame provided twice in this example so first one is being initialized using the workspace so this is to create objects inside workspace and the second one that you can see here it's being initialized as databricks account so basically this

8:22will be used to create objects on account level for example metastore right so and this you can see that uh we have given it a different Alias like Azure account and uh yeah and we are going to use like Azure key to authenticate uh in this case right anyways so so far we have just initialized the provider and just set some variables and then I'm executing one module so let me go into this module

8:45and I'll let you know later why I had to create this module separately so this module is going to create basically The Meta store as well as it will sync users and groups from aad to databricks account so let's go into this module top part is quite same you are just initializing some variables setting the provider and all those things once I have done that uh

9:08basically I'm gonna create uh the first meta store as we discussed earlier to create the meta store I need uh manage Identity or access connector so this is what I'm doing uh here once I've done that I'm creating a storage account that will be used to store the manage tables for The Meta store once I have done that I'm creating a container inside the storage

9:30account and once all of these things are done I need to also assign a role to this manage identity that's it that we created here to be able to access this storage account so basically if you have to manage identity uh permissions to read and write uh inside this storage account so the role needed for that is the storage blog data

9:53contributor rule once all of this is done then we can obviously create our uh metastore Unity catalog meta store as you can see here we are meeting with primary uh using the storage account here uh as a root storage and once I have done that I'm gonna attach this manage annuity to this uh to this meta store so that whenever we are uh we are we need to access the

10:16storage account uh through this meta store we have that permissions uh behind the scene working for us okay and after that once I have created The Meta store I need to attach this metastore to the workspace so that that workspace can become Unity catalog enabled so this is what I'm doing in this uh in this resource basically uh we have choose on the workspace ID The Meta

10:39store ID and just you know uh attaching The Meta store to the workspace okay and after this point I will start uh setting up the users and groups so I've just declared another provider uh you on the account level as you can see here I'm providing the account URL here okay so I had the the AED groups

11:02that I wanted to sync as a variable okay so I'm just creating some local variables uh local variable from from that group right uh and then I'm pulling some more information about those groups sorry groups from Azure active directory like we need more attributes uh to be able to work with those groups once I have done that then I will add those groups to databricks account right as you can see I'm using the provider uh

11:25here which is databricks account Azure account right so this is my provider which is uh using databricks account so I'm gonna basically Loop over these groups and I'm gonna add all those groups inside database account after this point I'm gonna start adding the members so when I talk about members I'm talking about users and service principles right all the members which are already there inside these Azure

11:49groups right okay so to do that I'm just creating some again some local variable which is just kind of you know uh extracting all the members uh from these groups after that I get some more information about these users because I need that to uh add them to databricks account and then in this uh in this variable I'm

12:13just extracting the groups I'm not extracting the service principle just the actual sorry uh let me correct that uh so in this uh variable local variable I'm just extracting the users not search principles because I just need physical users first of all and then once I have I have those users I'm gonna add them to database account using this database user resources as you can see I'm

12:35looping over all the users and just setting some username and external light you know all those things from from this uh from this local variable so this is where I create all the groups so users inside databricks account and same thing I'm doing for service principal so in this example I'm just extracting all the search principles and then I kind of create all those responsible inside databricks account

12:57okay so basically so far we have created a meta store we have uh synced different groups from active directory to databricks account and also we have synced different users and search principles from those groups to dataics account now lastly I'm gonna give some more privileges to these groups because as

13:20you remember I had a group called account Unity admin so basically this group will be uh the account admin for us so I need to do that and how I can do that basically I'm gonna as you can see what I'm doing is that I'm creating a new local variable which is extracting all

13:42the members from the group group whose name is account Unity admin right so this is a way to kind of filter between the groups and just get the group which is the admin and then we are just extracting uh the members from that group and storing that email sorry yes immersion I have just something you need to clarify if you can explain to uh the

14:04viewers what's the difference between like uh the account admin and data breaks like uh admin because sometimes they can have this misunderstandingo between those two types of uh okay so basically when we are talking about account admin uh so these are the you can like consider them like super users of database account they have a lot of privileges like they can create

14:27workspaces they can you know create meta stores and all those things so they have a lot of privileges but when you talk about just a workspace admin that user is the admin of a workspace itself they have privileges around workspace for example creating different clusters or making other users admins but they don't have privileges on accounts like they cannot create meta stores they cannot add other users as account admins right so when we talk about account admins

14:50it's like super user or detox account foreign yeah thank you so basically in this uh in this local variable my aim is to extract all the members who are in this account Unity admin group because all those members will become account admin and later we'll see all these members will also become owners of all the

15:14agility catalog objects that we will create okay okay so I've just uh got this list I'm just again getting some more information about these users and groups uh from from Azure and uh and then what I'm doing as you can see here that I'm using this data Fabrics user role uh resource which is making all these uh basically

15:38all these users who are part of this account Unity admin group as the account admin right so this is where we assign them the permission to become the account database account admin right okay I think this this all this module is doing you yourself do you have any question or can we move to the next

16:00section I no I think yeah you you gave so many details I think people betrayed thank you okay cool so let me just also quickly show you the output because this is important since this module is called in the main terraform template so basically we are going to use the outputs of this module inside the main template because this provides us a lot of different resource IDs that are going

16:24that are going to be required right for example I'm creating multiple maps of groups users and such principle which is kind of mapping their external ID to their ID inside database we need this ID like this is the ID coming from Azure and this is the ID inside database and we need this mapping uh later on to do some assignments same way I'm also returning the ID of manage identity which gives us

16:46permission to access the storage account I'm returning the returning the ID of metastore that we deployed workspace side and all those things and you'll see that these these values these outputs will be used inside this main template okay so so far basically uh we have deployed through this module The Meta store groups and users now we are gonna

17:10uh basically put the user sensors principles that are already have been added to database account to their groups so basically we need to uh we need to kind of add the membership right we need to make sure that those users and groups are now part of their groups because as you can as you saw

17:34earlier that when we created groups and users we just created them standard alone there was no mention of membership right because the membership is defined in ad but on databricks account so far we haven't done that so we need to do that part to do that we can use this databricks group member resource basically where you need to provide the group ID and the member ID and then that Association happens

17:56behind the scene for us right so as you can see in this resource we are going to Loop over all the groups then we are going to Loop over all the members inside each group and then we are going to just use the group ID member ID uh as the input of this this resource and we do this assignment and this is the reason this exact resource is the reason

18:18that I had to create this module why let me explain because uh terraform has a limitation uh when you want to use let's say an attribute of a resource inside a for Loop the value of that attribute should be known at the planning time terraform planning time for example the member is

18:40a members attribute of a group Azure group so when we are doing terraform plan we know the groups but we don't know the members it's not known at the planning time so that's not a problem if you are not using this inside a for Loop but if you are using inside of this inside a for Loop terraform doesn't allow this in the previous versions it used to work in the newer versions it

19:02doesn't work so to kind of work around this problem you have to kind of break your template into two parts first part is where you actually add your groups and users beforehand so this this part will execute uh Standalone and once you have done that then you can run the subsequent part that's why if if I can go back to

19:24my uh repo as you can see here in the section below to run this template you have to run terraform apply Twice first you have to run this up to this module which will just create metastore add users and groups to account and then you can run the rest of the module uh to kind of work around this problem anyways that's not a problem to be honest

19:46because if you are running this template using a CI CD pipeline then just it's like two commands that you you need to run in the cicd pipeline it's gonna be totally transparent to transferring to your user so that's not a problem but yeah that's still a kind of a limitation that's why I had to kind of create this separate module and run this this template into uh two different parts

20:08good all good Yusuf okay awesome so this so so far I have created a meta store I have added users and groups to the account and I have also uh added them like I've also uh kind of uh taken care of a membership of different groups uh users and service principles now what I'm gonna do I'm gonna use the identity Federation

20:31feature where I I'm gonna sync different groups from account to the workspace right to do that I'm gonna use this resource and you I'm gonna kind of loop over all the groups because I just in my example I'm adding all the groups to the existing workspace so I'm going to Loop over all the groups and then the workspace ID then the principal ID is object is the object ID of each group and one thing I'll also also like to

20:55highlight here is that if my group name is account admin sorry account Unity admin then that group becomes the admin of workspace otherwise all the other groups will be the user of that workspace right so this is how you are gonna also take care of who becomes admin of your workspace right cool and after this point I'm gonna

21:17create a Dev catalog as we discussed earlier so I'm just obviously creating uh some priorities prerequisition sorry prerequisites for that sorry I mean that word anyways so I need to create these these things beforehand uh I'm creating the storage container in the same storage account that we have used for metastore

21:40and yeah I'm creating a storage credential uh which is uh again using the same manage identity that we have created earlier because this is the same storage account so I don't have to create a separate uh manage identity but if I wanted I could have created separate one and you can see that in these two resources I'm definitely leveraging the output of the module because uh these two uh resources the

22:03master resources like storage account and the manage identity were created inside the module and I'm just using those uh the IDS of those or the name of those resources inside this main template right so basically I create this storage account or the container inside the storage account the manage uh sorry the storage credential which is just uh loop or sorry enveloping or uh encapsulating

22:26the manage identity and after that I'm gonna create this external location which is just pointing towards this container and using this manage identity as as a way to kind of uh connect uh to this container because we need some credentials for that right so so far we have set up this external location and now I can create my

22:48container sorry my catalog my Dev catalog so I have given it a name I'm using this uh storage uh root URL of this container as the root storage where all the manage tables of this catalog will be stored one thing I which is also important that I'm also setting the owner of this object which is again my account Unity admin group so that I know that all the

23:12members of this group by default and owner of this catalog right after that I'm gonna give some grants to different user groups so I want all the user groups to be able to use this catalog so I'm just giving them all of them use catalog grants using this databricks grants resource then I'm going to create some schemas or

23:34databases so I'm creating a bronze schema uh silver schema and gold schema okay for all the schemas again I'm making this group as owner of these schemas and then I'm also giving some grants like for example I want data Engineers to be able to you know use the schemas create tables inside that schema create functions execute functions

23:56modify table read all the tables because data Engineers need that access in my example uh for I'm talking about bronze layer here for silver layer data Engineers have same privileges as bronze layer for data scientists I'm giving them just use schema and select privileges I'm not giving any privileges for data analysts for single layer and for gold layer for data Engineers same privileges for data scientists and data

24:20analysts I want them to use this schema and also be able to select from all the tables from this schema so that's pretty much it in terms of terraform example Yusuf any questions comments no all good I mean it's like you're doing the summary of unity catalog and explain each component so that was super great and also very

24:43useful cool so probably I can quickly show you the real deployment so we already saw this part yeah so as I mentioned earlier like if you want to run this template uh just keep in mind couple of things that you need to uh like create these groups beforehand in your active directory you

25:05can keep the name as if you want to change the names of your groups that's fine in that case you would need to change some of the things in templates for example every time uh we are using uh you know a group name for example here uh uh in hard-coded so you will have to kind of change that but like for the sake of Simply Simplicity you can keep the name as is so that you

25:28don't have to make any changes okay and once you have done that and as we saw as we talked about earlier that you need to have at least one database workspace to be able to access the account console right then you can just run these templates and then you have run uh the terraform templates using the commands that I mentioned

25:50earlier as well and just like in addition the workspace should be premium not a standard yes absolutely absolutely catalog that's that's a requirement you should be using a premium workspace so yeah once you have created your uh like created uh one workspace you have created your groups inside active directory then you can just kind of Clone this template and run it as is if

26:12you want uh obviously you have to set the variable values but that's fine and then you can run these comments one by one or inside a cicd pipeline so after you run these commands uh you would see a lot of resources created on database accounts account sorry so I let me go to the database account so you will see first of all there will be a metastore primary and in the it's like in my

26:36example it was invest your operation you can obviously choose a different region and then you can also check out uh the storage location that you created for this um this meta store uh where you know uh the managed cables will be stored you can also see the workspace which is linked to this meta store and this is a workspace that you need to have before you run the template then you can we can

26:58go into this user management section so I can see uh all the groups are properly synced to the account I can even check that they have all the members inside them uh I can let's go here I can see that there's a search principle there's a user right so all the members are properly synced here and then what we can do we can also go check the

27:20workspace so I'm in the workspace here let me go to the admin settings I can see that all those groups that were in the account are also present in my in my workspace and all the users are also properly synced from account to workspace all the search principles are also synced from account to workspace

27:42and on top of that I can explore the data section where I should normally see the catalog and and the schema so here is my Dev catalog as you can see account Unity admin is the owner of this object it has all these schemas right uh let's go into details so you can see I'm using the the the container that I created uh using terraform inside the

28:06storage account as the uh the root storage location for this catalog and let's go into one of the schemas that's going to Gold so I have this gold schema obviously it doesn't have a table because I haven't created one let's go into the permission so I can see the permissions is that like all these users can like what all the different groups can do let's go into uh bronze schema

28:30and permissions okay so yeah just yeah I just wanted to highlight these these things so yeah uh I think that's that's all Yusuf I wanted to showcase yeah like how to set up and check catalog uh sync groups and assign permissions yeah I think yeah that was

28:52pretty well detailed and uh just keep in mind guys that every anyone can access this GitHub I will add the link in the description of the video and you can also contribute if you want to this uh repo as well you can also add dementia yourself

29:14okay so thanks thanks a lot for inviting me yeah that was it thank you for this uh uh this walkthrough or step by step uh getting started step by step in unity catalog using uh terraform so thank you and thanks thanks again