Terraform for Databricks: Automate Infrastructure & Governance

0:02hello my name is Yusef welcome to another video around configuring and deploying your lake house environment before we start as a recap we first discussed how to set up Unity catalog then we talked about configure scheme to synchronize groups and users from your ad today we'll cover how to leverage terraform to deploy your lake house

0:25platform and I have with me a scene who will guide us through awesome thank you you said thanks for having me all right so tell me more about the newly introduced Arab databricks terraform modules awesome so yeah what I want to share just before we start is just to go through the databricks styrofoam

0:48provider like if you go to the documentation you will see all the different resources that are covered uh with the provider a couple of things that I want to highlight some things that I really like about the work done by the team at databricks is first we have these guides so if you are interested for instance in deploying

1:11your databricks workspaces environments on any cloud provider you can just go through these guides and it will help you step by step on these deployments so regarding the models what the team did is if I actually search for databricks examples

1:33so as you can see it's a set of IIT plus 30 models and examples on how to actually configure and deploy your lake house environments on Azure AWS and gcp and the goal is to help customers and help everyone using the terraform provider to accelerate the deployments

1:57and build some reusable models that can be used for different use cases and also for different environments so if we have a look at this for instance you can see the different models that we have today for private link on Azure you have something on AWS as well and also on gcp

2:19so you have the models but you also have some examples on how to use these models so we have two options you can either just call the model and use them or you can start with an example to see how it works of course you can still configure your model if you want to actually add newer components in your new resources through your model

2:42now if I go as you can see it's covered with this repository terraform databricks examples if you go to this repository I can just scroll down and I will see the contents of these uh these models so you can see all the examples that we have on Azure and also all the example for AWS gcp if

3:07I scroll down a little bit further we have the same thing for the models not only the examples and one last thing to highlight is you have these models that can be used in your environment to configure deploy your environments but you also want to actually also automate your terraform deployments so you would see some

3:30examples for CI CD pipelines with a step-by-step guide on how to configure your pipeline so if I go to Azure devops for instance I can scroll down and you can see the workflow end to end and also a step-by-step on how to configure this and can you walk us like through how to

3:53configure a module to interact with a cloud environment okay that that's actually a great question to start with so if I go back to my uh Repository to answer your question it depends on which uh cloud provider you're using and which telephone provider you're using so

4:16in our case we will be working with two providers through telephone providers you have Azure and also databricks answer your uh question the goal is to go to the terraform registry and you would go through the for instance if it's Azure I will search for azure

4:40and you can go to the documentation and the first page of the provider you can see a step-by-step guide on how to configure this same thing for databricks so if we search for databricks I always do the same thing we go to documentation and you can see an example usage how to use the provider but what's

5:05uh what we're looking looking for is the auto Authentication so in authentication it will give you the different options to actually configure the provider so it can be tokens or service principle using Azure CLI it depends on your requirements and your actual setup okay and can you show us the different

5:28uh available modules and examples yeah so as I said before you can see all the models and examples either directly in the repository or you can go to terraform registry search for the model and then you can see these examples but just to give you a quick overview

5:51if I come here like you have on azure Azure databricks lake house uh same thing on AWS you have some models on Unity catalog on private link and so on okay that's I mean that that's super cool but let's go to the hottest topic so in my previous video I did show people how to uh

6:13configure Unity catalog can you show us the example about how to do it using terraform awesome so yeah just before we start like uh anyone watching the video I suppose they followed along with your previous videos on how to configure Unity catalog and also how to sync users from their

6:37identity provider either Azure ID or OCTA or another identity provider to the account console now we will use this this uh all this information to actually use terraform and automate the whole process for instance I want to deploy some catalogs I want to create first my

6:59metastore Define catalogs schemas and tables assign permissions and so on so to do that uh we have two things as I mentioned before so you either you can either use the example in our case it's ADB lake house that we will use in our demo or you can

7:22scroll down and look for the actual models that does uh the work that do the work so for instance if I'm interested in unity catalog I can go for Azure databrick Snake House Unity catalog and you can have a look at how to use it

7:44so in our example as I mentioned we will use this you actually have two options you can go to the example directly and as you can see you have like a guide on what it will deploy how to use it or I can also if I go back and go to registry

8:10I can look for ADB lake house and this is actually very useful because you have the readme of the repo that is describes everything that it does also you have all the inputs that you have for your terraform code the actual outputs and also the dependency if it's using another provider and as you can see it's using

8:34Azure provider now what I can do is if I want to start using this from scratch I can clone the repository in my local environments and starts using the example directly so this is what we will do we'll take the uh Azure databricks lake house example and we will see how to use it to deploy

8:57our resources okay so if I switch to my vs code you can see that I already cloned the repository I already have everything available for our demo and in examples I can go to Azure databricks lake house first thing you need to do is

9:19Define the variables needed by the model so if you go to variables you will see all the variables needed and what you need to do is create a tfvars file and give a value for each variable unless if it's optional so I already did this I defined my tfr's

9:44file second thing you need to do is configure the Azure provider the authentication for the Azure provider and also the authentication for the databricks provider and this is what we covered in your previous question yourself okay so once I do this uh in the main

10:06file you can see all the models that it's actually calling so it's calling the first model which is ADB lake house this one will create a new databricks workspace and also a new meta store for Unity catalog so it will create the workspace the

10:28metastore and attach The Meta store to the workspace next we have ADB lake house UC metastore this will create some uh the like all the resources needed for the for the meta store and also configure the owner or the admins or The Meta store so for example if you already configured your

10:52skin to bring users from Azure ID you can take a group of users and assign them as admins of the metastore and the next one is we will Define the service principles and groups that we want to add to the account console or to the workspace and the last one as you might guess is

11:14actually defining the resources within the metastore so for example our storage credential external location catalogs schemas and tables yeah it's much easier than defining everything uh using the UI exactly it's actually super helpful because first I can use this in different use

11:37cases different environments and also I can use this as scale if I want to make any change it's easier to just add the change to your code and apply it to all uh their infrastructure and it's also very useful to actually apply some best practices and enforce these best practices for example the naming convention the type

12:01of resources to use and so on all right so in my case I already run a terraform in its data from apply because it takes some time to actually make the change what I will do will go to my environments first thing if we go back to the example ADB lake house

12:24and you can scroll all the way down at this level you can see the steps that I just mentioned so first let's change the uh specify values for each variable in this new file second configure the remote backend if you need for example in a storage credential

12:46storage account on azure and then you can run terraform inits maybe a telephone plan just to list all the changes that will happen and then you can run terraform apply all right so if we go to the account console I can go to workspace

13:08and you can see that I already deployed a new workspace called TF demo and at the same time if we go to data you can see that I already configured a new meta store and the metastore is already attached by go to the metastore it's already attached to the workspace that we just

13:30created so basically with this model with deploy the workspace a metastore and attach The Meta store to the workspace and at the same time if I go through my uh if I go to my workspace and go to the data Explorer on the left

13:52you can see that I deployed a new catalog called bronze catalog Dev and you uh schema and a new table in this schema same thing we did for the storage credential and the external location and at the same time if I go to my Azure

14:14portal I can see all the resources that were deployed so like if you want to make any changes now for instance if I go back to this level and go to variables you can see uh for instance we have tags if we want to do something like chargeback

14:36and we want to tag all our resources using terraform you can just go again to the TFS Wars file you specify new tags for example the name of the environment the name of the use case the name of the team and then you run terraform apply and it will apply all the changes to all your resources if you are using the tags

14:59[Music] um and since you're mentioning this can you for example add a comment to an external location just using like by using terraform yeah so you can definitely do this yeah this is like a simple example on how to use uh terraform you can also uh suppose that you have a new admin that just

15:22joined the team you can add them to the group of admins and it will automatically assign them to the metastore admins and to answer your question what I can do I can either Define this if I go to main I can make the change directly at this level to add the comments to the resource that I'm interested in or what

15:47I can also do is since I'm using a model and I want this to be applied to all my resources I will go through the to the model that I'm using which is data assets and in UC data assets you can see that we have databricks external location

16:10so in databricks external location we have an attribute called comments and I can easily change this let's call it just created by terraform just to show you that the change will be propagated to the to the workspace first we will go to the account console sorry to the workspace

16:32and if I go to my external location as you can see I created an external location for the landing Zone and the comment is created by terraform so this is only one external location one workspace if you have many to deploy with terraform it will be easier to scale now if I go back and

16:56I save these changes and I will run terraform apply so let me search for apply all right so if I close this so with terraform apply it will show you the changes that will uh will be made to your environment and in my case I use

17:19Auto approve I wanted to just go ahead and deploy the changes but normally if you don't specify this it will tell you the changes and then you can confirm and apply these changes to your environments so while this this is running I just want to show you one other thing if I go to the databricks provider

17:44since we are talking about Unity catalog I can search for Unity catalog and you can see all the resources that can be managed by unity catalog so in uh for instance as I mentioned before you want to apply some permissions on your data assets to your users groups and so on

18:06you can go to databricks Grants and I will scroll down just to show you an example on how to do this so on the right you can see which Unity catalog objects you're concerned with either it's a schema table view or

18:28catalog or other assets suppose it we are interested in catalogs we want to manage permissions on catalogs so as you can see super straightforward I Define the name of the group of or the users and the permissions that I want to give them on this catalog

18:51and this will be scaled and applied to all the catalogs if you want you can iterate actually iterate on all the catalogs and get these permissions if you want all right let's have a look to see if it will spare you a long long time that's for sure yeah definitely so as you can see we changed only one thing we didn't

19:13create any resource we didn't destroy any resource only did it updated the external location with the new comments now what we can do is again go back to the workspace and you can see that the comment changed automatically this is only for the comments you can do the same thing on the owner the

19:36credential oil itself the storage credential itself and all the other assets and there are objects in the workspace that's pretty useful and what if I don't know tomorrow I want to contribute or maybe report an issue what are the steps that I need to follow oh awesome thank you for asking that so

19:59if I go back through the Repository it's actually a great question because like this is uh efforts from the team at databricks to create new content on our new models and examples on terraform but anyone can contribute and create new content so for instance if you have

20:22some examples on how to deploy like databricks workflows or any use case with terraform on databricks either on gcp Azure or uh even on AWS you can create a new pull request like you can Fork the project to your gits

20:45add your content and then create a new pulley request to add your contents to the Repository and once you add the content you can include it in the readme for instance you added some examples on Azure you can specify the examples and just a description to explain what you did and if there are any issues you can do the

21:08same thing you can go to issues and just report this issue or directly create a pull request to solve the issue okay and one more question this is my very first time with terraform can you point me the resource that I need to check in order to be let's say a terraform expert awesome so yeah I can see that you're

21:32being confused with what I was saying so uh if if you are new to terraform you want to learn and how to use terraform there are a couple of resources very useful resources and trainings I think my favorite is the resources provided by ashikorp and there's actually a study guide to prepare for

21:55the terraform certification and it will provide you with all the useful resources for terraform like step-by-step guide on how to use terraform how what are the best practices on around using terraform and so once you cover all this content you can actually uh if you are interested you can pass the terraform certification

22:19okay okay that that's super super cool uh any other thing to add to our viewers before we close the video no I think that's pretty much it once you deploy all these resources you can play around and your resources to your terraform code just to try it out and if you have any interested interesting

22:43examples feel free to create a point request and add them to the Repository okay thank you thank you so much yes in so I'll take all the links you did share with me and I'll add them uh in the description of the video and if you have any questions feel free to reach out to us in the comments we will make sure to answer all your questions

23:05thank you for your time thanks Yusuf thank you bye