Delta Sharing Behind Firewalls: Private Endpoints Proxy Serverless NCC

0:04Hey folks, today I'm with El Shakun, specialist solution architect at datab bricks and he will help us understand how we can configure data sharing protocol when the workspace is hidden behind a firewall. Hi Ali. Hello youf how are you doing? Introduce yourself and yes I'm doing very very well. Great. My name is Enra Benikun and I'm what we

0:28call specialist solutions architect. So I I mainly help within the organization the solutions architect to deal with security cloud infrastructure or also cloud architecture challenges for our customers. So I mainly help customers on those topics and today my goal is also to explain you how to deal with security challenges when working with data

0:51sharing. My first question is can you maybe introduce quickly what's delta sharing? Yeah. So delta sharing is an open protocol first. So delta sharing is composed of three main part. First part data sharing server that implement open APIs. Second data sharing client in

1:13which you have the possibility to read the data exposed by a data sharing server within any kind of platform. And third, a data sharing protocol in which we specify APIs in order to make sure that we have the possibility to share data from what we call a consumer to from a provider sorry to a consumer without having to replicate this data

1:37and all the capabilities of this open protocol is mainly based on non-replication of the data. So within the datab bricks platform, you have the possibility to share any kind of tables, any kind of unity catalog objects without having to uh replicate your data by using some specific protocols or specific technologies of the of the

2:01cloud provider for example on AWS by relying on uh preigned URLs for example. Okay. Which means I can share tables, volumes, models, maybe federated databases. Exactly. That's amazing. And do I need to have a compute always on to share the data? No, definitely not. Basically, all the traffic is first

2:23built by using what we call creating a bu a trust between trust relationship between what we call a provider and a consumer. So first what you will have to do is as a provider creating a share and then specify this share the target of this share. This target can be on datab bricks another datab bricks workspace

2:46within another datab bricks organization or within the same organization but in another cloud region or you also have the possibility to share data to any other kind of clients. In that case for the first approach it will be what we call datab bricks to datab bricks sharing and when we want to share any kind of datab bricks asset for example

3:09table to other consumer in that case we call it as datab bricks to other okay and my main question is like okay you talked about pre-signed urls but what happen if my storage account is behind a firewall how can I how can I benefit from the delta sharing protocol yeah Let's go into more in details into this diagram to understand what's

3:33happen under the hood when uh when you try to access to a shared table within in data sharing. So as I said as a data provider I have the possibility to share the data to a data recipient. In that case uh trust relationship is built between the data provider and the data recipient. Then when I want when the

3:57recipient or the consumer want to access to some data, the client will first just try to query the shared table by doing a request. First the request will be controlled by seeing if the user has

4:19the right permission to access the data and also is authenticated. If it's the case the request is audited and saved into audit tables. After that the delta sharing table create what we call a pre-signed short-lived URL. Then

4:41after that the delta sharing client has the possibility to access the data that is shared to him directly by accessing the files in the object store. Meaning that when I want to access to some data that is stored within an object store, the communication that you will have to set up is between your delta sharing

5:03client and the underlying object store. In that case, if my object store, for example, my S3 bucket or my storage account is behind the firewall, I will have to create a private endpoint, VPC endpoint, whit list the subnets and or the VNET of my client in order to access to the underlying storage and access to

5:26the data. And how can I implement this? Okay. So in order to implement that I will share with you a diagram that will present you practically in terms of networking how we can implement that. So if you share if you see my screen here you will see that uh in order to access to the data first you have the

5:48possibility to directly as a consumer to access the data directly. So as I said in order to access to a share table the consumer will need first to request the data to the delta sharing server and unity catalog server in order first to authenticate the user check the

6:10permission and then generate the pre-signed URL. If your compute is part of the datab bricks workspace standard compute in that case what you will need to do is just to set up a private endpoint directly into your storage. If you have different datab bricks compute within different vinets, you also have

6:32the proxy for possibility also to set up a proxy that will be the central part of your layer in order to make sure that you only have to specify one private endpoint and all the traffic is passing through this proxy. Then you will have just to set up as many private endpoint as you want. If you don't want to set up

6:54a proxy, you will not have to create one private endpoint per client. For example, if I have multiple workspaces, in that case, I will need one private endpoint per each workspace. If I want to access that only through an appliance, for example, a proxy. In that case, I just have to specify only one proxy, one private endpoint for the proxy. And in that case I have to

7:17specify within my client some uh spark configuration in order to proxify all the requests to pass through this proxy. If you want to use serverless you also have the possibility to configure what we call network configuration connectivity and the setup is exactly the same but at the datab brick

7:39serverless level. So in that case your compute plane will be a serverless compute plane. you will have to create an NCC that will be within our environment and that will access directly to the underlying storage that is associated to your shared table. Okay, I think you made you made things clearer now. And I guess folks, if you want to deploy or benefit

8:02from data sharing when you are using uh VPN like um a private network for your workspace, you know what to do. And I will we will add the documentation in the description of this video. Anything to add? To be honest, I'm really really really excited by uh first all the

8:24implementation that we are currently doing using delta sharing and I'm pretty sure that we have more and more also excitement exciting announcement to come with delta sharing to make sure that our protocol is more open but also more performant and open to all to make sure that the networking configuration is easier to set up when dealing with data

8:47sharing. So yeah, keep keep yourself posted for the future announcement. See you.