Debug Spark Locally on Databricks: Connect V2

0:01hello and welcome to this session my name is Yusef and today I have super super great news for you databricks has announced the public preview of dbconnect V2 which will allow course developers to use the power of database from any application running everywhere and what's dbconnect so it's client

0:24library for databricks front types which will allows you to write jobs using spark apis and of course random remotely on a database cluster instead of the local spark station and when can I use dbconnect so can use it to run a large spark large spark jobs

0:46from any python application where you can of course import a pi Spar you can do step through and debug code in your ID like when even working with a remote of course cluster and you can interact with link When developing libraries and the question how can I get started what are the requirements so you can you

1:10need to have a databricks runtime 13ml or databricks runtime 13 and it supports only Unity catalog clusters it can it either includes single user or shared cluster access mode and something very very important how can I uh install uh

1:35databricks connect client so it's very easy just need to go and install this uh databricks connect 13 and when it's done you need to do some other configuration so first you need to go and get the uh the hostname so for my case that would be my hostname

1:59that I need to add over here in this in the in this database session then I need to get a token and to generate a token I just need to go to user settings to generate a token and of course ID and I will copy this uh this token over here and then of

2:21course databrick cluster ID and for this just need to go to compute and create on this cluster and then copy okay this string over here and then paste it and for for my for my example I I'm creating this data break session and

2:44from there I will just go and grab this Titanic P1 which is the which is a data that I have on my database environment it's called uh Titanic V1 and with this Titanic if we want I will do some uh I will create a new column uh called Ace transform where I would do like if it's

3:06the age is lower than 26 then it's uh Young if it's higher or equal 26s versus old I will add the status whether this person is dead or alive and I will store I will store this table so now I will just run this uh run this code and it

3:28would then of course I will save this table and then based on this table I will run other aggregations and then save it as a Titanic course uh second uh table called Titanic aggregation so you see step what is done step two is done so now if I go to my you see that's the new tables has been created so if I

3:50click on aggregation I would still benefit from the lineage because I'm using let me see Titanic P1 was used to create Titanic filter and Titanic feature was used to create Titanic aggregation and of course it's very useful because here the this dbconnect library is not dependent on Visual Studio or any other ID which means you can use it with all your favorite IDs to

4:12run your code and uh and benefit from the compute of databricks you can add them on applications and if somehow your cluster is off when you run your code then it will trigger uh the cluster and the cluster will will uh will start so make sure to follow the databricks

4:36blog to get the latest news and see you