Firewall-Friendly Databricks: Secure Architecture & Private Link Setup

0:04Hey, today I have the pleasure to host Rali Bun, specialist solution architect at datab bricks and his field is security. Hi Ali. HelloF. How are you doing? Good. Can you first introduce yourself? Yeah. Uh my name is Ali Ben Shakun. I'm what we call specialist solutions architect. So um I mainly work

0:26with the solutions architect team at datab bricks and my specialty is more uh focused on security cloud architecture and cloud infrastructure topic within uh datab bricks organization. So I guess your as you mentioned your field is security. Exactly. Okay. I understand why you go to the gym. I wanted to make made to make this joke. Okay. I have a

0:50first question for you. um what are the best practices when it comes to deploying uh an Azure uh datab bricks environment uh behind a firewall because I know this is a tricky discussion and uh you need to do some configuration to make sure that it works properly. Exactly. So basically before uh talking about the firewall let's understand

1:12what's the current architecture of a datab bricks workspace. So if we go into if we go through this uh architecture that I share with you, you will see here that we have an architecture based on Asia but that's exactly the same on AWS and GCP and the datab bricks architecture is divided into two parts.

1:35The first part is this part this one called control plane and this control plane is the main orchestrator of our platform. So the control plane is uh one of our uh uh platform part in which we can first uh host what we call uh unity

1:58catalog. So, Unity catalog is our main authentication and authorization uh engine that will help you to uh provide to first decouple the uh compute uh part from the storage part and then also access all to all Databricks assets uh within the platform uh in a fine

2:22grained way. Uh a second part that is also hosted by the control plane is the web app and also the orchestrator that also help you then to orchestrate the compute and this is the second layer of the datab bricks architecture. It is what we call the compute plane and basically within databicks platform we

2:45have two layer of compute plane. The first flavor of compute plane is what we call the standard compute plane or the compute plane within your VPC vinet. And in that case the control plane orchestrate the deployment of VMs within UVET subnets in order to access your data. And we also have the serverless

3:07compute plane uh in which we also orchestrate the deployment of VMs but in that case it will be within our Azure AWS GCP cloud and tenants in order to access your data. Basically the compute plane is uh the layer in which we access all the data. Uh so basically when we

3:31run some queries within the web app, it first sends the query to the control plane. The control plane translate this query uh as a spark job to uh the compute plane. The compute plane executes this spark job in order to access the data uh into the storage layer and then do the computation and

3:54send back to the control plane. And when we say how to secure uh every layer of our three-level architecture, we have to take into account three uh connectivity. The first connectivity that we can have is uh the user access basically when I'm a user and I want to access to the web

4:16app that is hosted within the control plane. The second level of security that we need uh also to secure within uh the databicks platform is how we can handle the connectivity between the control plane and the compute plane. And the third layer that we need to secure is also the security between the compute plane and the data plane. Basically

4:39between the compute within your your VMs and the uh underlying storage that we need to fetch. In that case, what we need what we will need to do is always to create private endpoints on Azure or VPC endpoints on AWS uh in order to access the data and also to secure access to the workspace. That

5:01sounds good. Um in in terms of uh connectivity let's first focus on the third layer meaning the access of the uh compute plane to the data plane. In order to do that basically the only thing that you need to do is to deploy your workspace uh with two or three subnets depending on

5:25how you want to access your data. Basically the best practice should be to deploy three subnets. one subnet for host, one subnet for container in which we will deploy all your uh datab bricks VMs and clusters and then a third subnet called private endpoint subnets and this private endpoint subnet will uh host all

5:48the private endpoint needed in order to access other Asia services or onremise services. For example, if I want to access to Unity catalog data within a storage account, what I need to do is just to create um uh a private endpoint in this private endpoint subnet in order then to access uh the storage account

6:11and in that case the connectivity will be done between the control the compute plane and the data plane. And I have one question uh for the subnets they they need to be need to make sure that you have enough slot for all the all the VMs and what if for example I chose the wrong uh number of uh slots can I drop the vinet and create a new one associated or once it's done you can we

6:35you can no longer undo it. Yeah. So basically right now uh you will have to redeploy your whole workspace but uh we have a private preview right now uh that will be soon also uh deployed and also promoted as a public preview in which you will be able to selfs serve uh the

6:57capacity to change UVET to change the subnet or to change also uh the subnet size associated to your workspace. Oh, that's amazing. Good news. Good news. And even if you are not able to uh deploy a lot of clusters uh at the same time because you have small subnets, in that case uh the best solution should be

7:22to use the serverless compute plane. Why? because in that case uh you don't have to worry at all uh on the numbers of uh concurrent clusters that you run within the serverless compute because in that case the access will be done directly from our infrastructure to your uh data layer basically your subnets. Okay, makes sense. Okay, second part

7:45that we need to talk about is the access to uh the workspace and basically the user that is within his LTAP in order to access for example the web UI or also the API. In order to do that, two way to implement that. First way to implement that in a secure way is to have within

8:08your Asure environment an express route on Asia or AWS direct connect on AWS or maybe just simply um a VPN between your LTAP and your Asia environment. In that case, it will help you to have a private private connectivity between your cloud environment and your LTAP.

8:30Then you will be able to access the front end, the back end private endpoint and the browser uh authentication private endpoint that will handle the connectivity between the user and the control plane. So basically if I zoom out you will see here that when I'm a user the network flow will be the following uh

8:53the uh the network flow will go through first the express route or the VPN connection if I'm au not authenticated in that case network flow will pass through first the browser authentication private endpoint that will handle the connectivity between the control plane and your Azure entra uh ID then send

9:15back the connectivity again to your environment in order to access the endpoint associated to the web app that is handled by the front end private endpoint. So it will then uh uh send back to you to the front end private endpoint in order to access to the web app and get back the brows in the your

9:37browser the front end application. Then if you want to access to any API uh within any other application that is hosted on premise or within your VM, all the access will be also secured in what we call the private uh uh backend private endpoint that we can also call it as data plane to control plane

10:01private endpoint and in that case it will handle the connectivity between the user and the control plane but also the connectivity between the compute plane and the control plane. So basically when you want to secure the connectivity between the compute plane that is within your subnet and the control plane you also have to create a private endpoint

10:24associated to uh the backend private endpoint. Okay. and uh everything you mentioned so far and when it comes to do like or to deploy the platform uh what's your recommended way is like for example for the first time should I just deploy the um the platform manually and

10:46configure this or just use terraform uh like from the beginning like what what would be your recommendation here? My recommendation is always to follow the best practices and in order to do that I highly recommend you to take a look at this uh great datab bricks repository called Terraform datab bricks sr. Srra

11:11stands for security reference architecture. This is u a terraform repository that is maintained by the uh datab bricks security community and that provide you a template to deploy your workspaces within AWS as GCP and AWS gov

11:33cloud. So my uh uh recommendation should be to visit this repository, have a look at it and also use this template as a baseline in order to deploy your workspaces and make sure that you deploy your workspaces with the same fashion with the same configuration and uh in a

11:56redundant way and uh with the capacity to you to deploy that at scale and you will find the link of this of this uh repo in the description of the record the video. Exactly. And and I have another question. Do you have any uh environment already deployed with uh with all those like behind the firewall and with private link and all the

12:19configuration you highlighted? Exactly. Yes, I already have it. But first, let me show you how we can deploy easily a workspace with all the right configuration within the VM. So if I go into my Asia portal and then you have the possibility to specify some first configuration basic configuration. So you can create your own resource group. For example, I can specify it as uh uh

12:45this name uh workspace RG. So it will create for me a new resource group. I can specify uh a workspace name workspace then specify the region should be in uh EU uh west Europe. I can specify a

13:08pricing tier. Please be aware that when possible, use a premium pricing tier in order to make sure that you will have all the security uh features within the premium pricing tier instead of using a standard or trial uh pricing tier that won't have all the security uh features that uh we want to use in our

13:32deployment. And then you can also specify a manage resource group uh in order to deploy your workspace. So I can specify this name as this one. After that we have the networking part. So in the networking part what we will have look at it is first if we want to deploy an Asia datab bricks with what we call

13:56secure cluster connectivity. So as I said secure cluster connectivity is the channel that we make between the compute plane and the control plane and in that case the best practice should be to deploy that with no public IP and with secure cluster connectivity enabled. Then I will also specify if I want to

14:18deploy my workspace within my vinet and I can say yes and in that case I will have to specify the virtual network that I want to use. For example, I can use this virtual network and in that case I will have to specify the public subnets that I want to use. For example, uh public maybe host subnet. Specify the

14:41IP range associated to it slash 24. Same for the private subnet. I can specify it as container-subnet. Specify the IP range associated to it. 0/24 then I can specify also if I want not gateway but it's not recommended if you want to route all your traffic that

15:05need to go through internet I highly recommend you to f to uh route all your traffic through the firewall and then you have the possibility also to specify if you want to allow public network access. If you want to allow public network access, it means that you are not uh mandatory using the front end private endpoint. Otherwise, if you want only to use the front end private

15:28endpoint, you should disable the network public access. And at the end for the network security rules, in that case, you should specify that as no Azure datab bricks rule. Then you can also specify the private endpoint that you want to create during your deployment. So the first private endpoint that you need to create is this one. So basically I can go and create my

15:51first private endpoint. For example, this private endpoint will be for backend private endpoint. So I can call it as a EGB backend private endpoint. The sub resource type will be databicks UI API. And then you can also specify the vinet in which you want to deploy uh

16:13this uh uh this private endpoint and also the subnet in which you want to deploy this private endpoint. You also have the possibility to create a private DNS integration associated to it in order to make sure that the private DNS zone associated to your subnet is well created. And that's it. After that you

16:35have the possibility just to click into okay and go into the encryption type part. For the encryption part we can uh check that after but you have the possibility to specify also encryption configuration in order to encrypt your manage disks manage services and also the workspace storage account. And finally after that you have the possibility just to review and deploy

16:58all your workspace. That's crystal clear. Thank you so much Ali for showing us the different steps we need to follow to make sure that we deploy um uh major Databricks behind a firewall. We're using all the features you have managed including private link and and all the other uh features. Thank you so much Ali

17:20and see you for our next series that will be about data sharing. Thank you. Cheers.