Databricks Connect + Serverless Compute Setup

0:04hey I'm USF and today I'm going to talk about DB connect V2 which is a feature that data Brees has released one year ago it's a client library for data Brees front times that allow you to connect popular IDs notebook servers and other custom application to a data cluster it enables you to write code using spark apis and run them remotely on a data

0:27breas cluster instead of local spark session it's built on the top of spark connect which introduces a decoupled client server architecture for Apache spark that also enables remote connectivity to spark clusters using the data frame API since the release one year ago data brakes has added a lot of improvement like full data support streaming support

0:50support for all pandas API and also Scala support and recently databas unveilled the serverless offering and now you can use our serverless offering with DB connect and that's the purpose of this session I will show you how to modify the configuration of DB connect V2 in order to benefit from the serverless

1:12compute or if you haven't installed it yet I will also show you how to run your uh setup quickly let me just share my screen and it will be good to go here we go the first thing you would need to do is to install the uh data brakes connect package so just run pipe

1:34install data brakes connect if you already have data break connect you just need to run an upgrade to make sure that you're using the latest version or you can just test which version you do have pipe show DB connect you you'll see that in my case I have version

1:59once uh the client or the package is installed you just need to run this command which is data breakes at login then configure cluster and host and of course I would need to copy the URL of my workspace I would choose a name for my

2:22case I will call it for example the profile would would be staging you would be prompt to uh to the uh the cluster just need to go and you'll be set up then you would need to search for the cluster you're going to be using for my case I'm doing the setup like it's the first time and here we go the next step would

2:45be to go to check just to be sure that the configuration is done so you need to open the config uh to you need call data brakes CFG ah my bad okay and you will see that if I go

3:11below I have my profile called stag in you see I have the cluster ID already there and the host so now if I want to run the classic architecture I would go to staging and run this in an interactive window it will connect to the classic architecture and since my

3:35cluster always on I won't need to wait for the uh spin up of the cluster what if you want to switch now to the serverless offering you would have two options you can remove the profile and just SW switch it with serverless and then you can run again your compute I can for example

4:08aware team is not null and I can run it again because you can see here I have not null so I can run this run Transformations that have the result immediately the second option you can have is to if you go back back to the

4:30configuration file and instead of having the cluster ID you can select an option from the configuration which uh use you uh let me show you an example okay I will go to stag in and then I would need to remove the uh cluster uncore ID and replace it with this

4:53configuration which is serverless computor ID equal Auto make sure to remove the ID and then you can just quit the uh configuration file and now if you want you can go back to the profile and switch again to stag

5:15in and that's it you're Now set up to use the data breakes connect with the serverless compute thank you for watching this session