UCX: Assess and Migrate Your Databricks Workspace to Unity Catalog

0:01hi my name is Yousef marinick and today I have a very special guest this guest is William and he will introduce ucx our Unity catalog migration toolkit hello youf glad to be there my pleasure so please let's let's

0:24be let's say straightforward can you explain quickly what's ucx so ucx is a tool that allows you to migrate very quickly into Unity catalog the goal of this tool basically is to guide you to the journey of the UC migration that may be a little bit complex for some

0:46customers yeah indeed indeed you you are you're right so uh what we're gonna do today what we decided with uh William that we got to show you how to install all the requirement tool required tool we're going to run this ucx on uh our workspace then we're going to

1:08explain the different steps you need to follow in order to uh have a successful migration and as you can see over here we have the uh code available in uh in GitHub so it's just this it's a data brakes lab project so it's go uh GitHub SL datab brick slabs SLX so make sure

1:33just to copy this link and for what's required you need first to have this is very important a python version 3.1.6 at least this is very very important and then once you have this uh at least this python version make sure to install the data break c as you can see in uh in the screen we need to

1:57install the the data bre library and datab braas CLI and then run this datab brakes configure within this datab braak configure you would be prompt or you be ask to enter your workspace URL the uh access token or your email and password once uh those steps are successful you just need to go to the CLI and of course

2:23open git clone the repo I will put in uh in um in the description of the video and then you enter in the go into uh ucx doc document and then you run this U sorry you run this specific command which is uh Point slash install sh that

2:46will run the uh the unity catalog script and of course it will take time yeah exactly so basically What's happen during step is install ing the script into a databas workspace so it's very important that first of all you configure the authentification from your

3:08local machine to a specific databas workspace because what we want to do there is simply to migrate one workspace into Unity catalog so you have to configure that okay and so the first step is do you want to open the configuration file in the browser or do you want to do it locally so I think you you want to do it

3:31in the browser right yeah I will just hit enter Because the default answer is yes it will automatically pop up a new uh Tab and you will see that it will open a yaml file where I will have the uh some uh configuration let's let's this um this

3:54configuration file just to explain very quickly what it does it just allows you to specify for for example what Warehouse is going to be used to run all of the database commands if you want to migrate only a specific part of the pass of the workspace for example you have also a possibility to uh do some multi

4:16threading so you have the N threets equals to eight by default but you want to increase that if you want bigger performance and you have also other configuration parameters by yeah and just for information because I already run the script that's why it didn't ask me this information but by the but if when you will run this for the first time it will ask you to choose which SQL warehouse and the path and the Treads

4:40and all the de and but again everything is declarative it's so simple and straightforward and again it will ask me open the job overview read me of course that's a yes it will again open a read me and this R me is very very important and William will explain every everything in in few minutes just just

5:03be patient be patient and you see now everything the insulation was completed successfully and now I just need to go to the r me and at the same time if I want I can go to workflows uh William please if you can explain quickly the uh all the workflows that was created over

5:27here exactly so all of the workflows that you see there are documented basically in the R.P that uh that has been already created for you so um what uh what this workflow does is they have basically multiple responsibilities the first one so the assessment is basically

5:49generating an assessment of all of the assets that needs to be migrated into Unity catalog because um you need to migrate the you need to migrate the tables you need to migrate the mount points you need to migrate the Clusters you need to migrate some pip lines into Unity catalog so there is some steps and

6:12these steps basically is just going to build a dashboard that is going to give you a view of all of the assets that you have in your workspace for these assets which one can be migrated into Unity catalog or not and and while I'm explaining that you say this um opening the dashboard the resulting dashboard

6:36this ass and just something I need to add so those those workflows will be created for you but of course you will need to run this uh this assessment this when you click on this then it will once it's done it will automatically create a dashboard for you it's called ucx assessment and then you will see what

6:58William was explaining so so the mic is yours exactly so as you can see this assessment gives you as I say an overview of what's inside of your workspace so in this in the workspace that that USF is using he has uh 650 tables 83 databases and you will see

7:23uh a first widget that explains you the table types so if it's a Delta table and also if there is a possibility to uh to upgrade uh to upgrade this uh this table or uh or not then you have a database summary basically in the database summary if you have uh assets replication required it means that you

7:47will need some manual steps to do to migrate this database but there are some of them that can be migrated with an in plac syc which is just one command and everything is migrating into UC so that's and that's this syn command uh is very uh is a very easy tool to use I

8:09will also add the link of the video that explains how to use the sync command to upgrade your tables from and tables and schemas from hi metastore to Unity catalog yes and then in the bottom you have other uh other assets so you have the mount points with which m point do you have in your workspace and you have

8:32also the external location that the assessment job is going to guess for you based on the external table that you have created right so this is very helpful for you migration then you have the Clusters if there is a cluster that can be migrated to UC or not in USF case everything can be migrated but sometimes you may have an old version that is not

8:54compliant with unity catalog then you have the same for uh for the jobs so uh workflows you have the same pipelines and you can see an example of where this cluster is not compatible because it was a dbr 11.1 h b basically it was uh for Scala so why because things has changed we no longer have uh standard and hien

9:18currency now we have single user and shared clusters of course we recommend mainly to use shared clusters and if you have some limitations then you can switch course to single uh single user exactly and uh so yeah that's that's pretty much it then you have pipelines you have service principle basically all of the assets that needs to be migrated

9:42to Unity catalog right this gives you an overview of everything that needs to be migrated okay so um then as you may remember there are other workflows in the in the picture so we saw the assessment and then there is also the workflow yeah this one workflow number two which is apply permission to backup

10:06groups then we have replace workspace local groups and uh we have also other ones so uh workflow number two three and four basically what they are doing is they are creating backup groups because uh workspace local groups needs to be migrated into Unity catalog so first we

10:28create back groups in case something goes wrong when we migrate these groups into in catalog we create a backup just to be sure so we create those backup groups we apply the same permission on these backup groups once this is done we replace the old groups okay with what we call the account level groups so

10:50basically we just delete them and recreate them by using the ion groups and once this is done we apply the permission to the ion groups that's work for number four and then once this is done we remove the backup groups because well everything has been done there may be some cases obviously where some failures

11:12may happen we handle this failure in the sense that we are not going to migrate a group if we have for an example a failure in the groups I mean migrate delete backup if there is a failure in the GRS obviously so these steps is already fully automated and you just need to click on the button to M that beforehand

11:35you need to well do everything by hat which is very commercial later on we will be able to have the same thing for the tables for the Clusters for the workflows for the pipeline for the service principle for the init scripts and all of the things that needs to be migrated into Unity Cog um hopefully all the features will come through yeah so and just to

11:58summarize UC experts will run an assessment to understand all the tables that can be upgraded easily and how uh they can be upgraded and of course then you will use the sync method to be able to upgrade the tables from hi metastore to UC of

12:20course you you can use sometimes deep clone you can recreate table you can using setas you will have like multiple options but really it's straightforward when it comes to mount points they are uh replaced by of course external locations and for with volumes as well where we can put for example your your

12:42your init script your jars or all the other things and the idea is really to help you do this in a very simple and smooth way and of course always keep in mind that Unity catalog will help you govern your dat data tabular and non-tabular data easily and will also unlock you more features like system

13:06tables like Lakehouse monitoring and more and more features so it's really your time to upgrade to Unity catalog thank you William for uh those clear explanations and and one more thing if they if a customer face a bug what are the steps that needs be out so

13:30first of all he needs to subscribe to your channel otherwise he's not going to have assistance and second of all he's kidding he kidding and second of all he needs to go into the ucx repository and file an issue so an issue is just like any standard uh giab repository you go into the issues and you just create a new

13:52issues and you explain very clearly what are the issue that you have and we are going to uh fix that eventually perfect thank you William thank you again for being a host no being my uh not my host your guest your my guest exactly I'm the host thank you and have a nice weekend

14:16thank you youf right