Migrate Hive to Unity Catalog: SYNC & DEEP CLONE

0:01hello my name is Yusef mourini and today I will show you how to upgrade your high metastore table to Unity catalog remember in my true previous video I did show you how to set up Unity catalog and how to synchronize your users using skim

0:23so today I will show you how to upgrade your tables and to do so we will use a command called Sync has been developed by databricks as you can see in in this article that I will that I will share with you as well and to do so the first step is really to go and uh

0:46uh set up your external location where you are storing your tables as an external location over here so I will go to data and then I'll go to external location

1:08I'll create my location I'll call it like external then over here I'll look for my from where the data is coming like where where this Excel data resides so for my case it's in ABF so my container is called external

1:30and this is my ADLs chain two TFS of course I already defined my manage identity and I'll create this external location first let's just do adjust it works I will Branch my

1:54self the permissions are privileges privileges and now I'll go back to my uh my notebook so I will use the catalog half metastore where I have my delay where I have my legacy table so I will show you all the schemas I have there so

2:17I have the crypto the football leagues and fujiks so um football fine so what I will do I will just use like football leagues and I will run it describe extended to show you where the data resides and of course the schema and everything you need to know so you see the catalog

2:40is high metastore the external provider is that of course we can migrate all the tables like Delta parquet and so on so the first thing I will do is really create the catalog and create a schema so for my case I called my catalog upgrade and then I

3:02created this schema called football weeks so I wanted to keep you know the same schema in the same schema name in high and unity so I called it food football mix as as I had in high metaster and now what I will do I will run this thing so and of course I can sync a schema including which includes all the

3:25tables that you have inside the schema from hive to Unity and to do so you just need to write C schema of course the destination which is my catalog database from the sources high metastore football leagues and you add the dry run so to dry run will tell me whether it will be a successful or there will be

3:49an issue so here I had this dry run success which means if I have if I uh if I remove the driver that will upgrade my my table and of course I can have several uh several codes so you can see over here dryer on success DFS root location external table is manage location Hive

4:11search so we have a bunch of errors and how you you can think about solving uh this issue so over here I will just remove the dry run syncing tables and it's loading in the meantime

4:34remember I had also database called crypto so you see everything is good so if I go now to this data Explorer I go to upgrade I have the football leagues of course I have the Moroccan players external and the statistics external that were moved

4:56of course you can see here the owner and by default the person who will run the same command will be the owner uh of this those tables but you can also Grant like uh set the ownership to another to another person uh if you want

5:18so in the second step I would also like create the schema called crypto and what I will do over here I will sync this scheme and here I have an issue you see I have this called deep dfdbfs root location it tells me that this is a managed uh High meta store which means the manage when the table is managed

5:40within High meta store it's not supported then how you can do this so you will if it's a Delta table you can use this like uh deep clock so I created a BT Bitcoin USD you see and I deep clone the BTC USD this one and if I run this command you will see

6:02that would work and now this is how you can migrate your managed uh high Met Authority and of course if I run again this uh this uh scheme I will migrate the other table without uh without an issue so the status class

6:24is a success so if I go to data Explorer crypto I I have this BT uh sorry this uh VTC USD external that was created and of course if I go to the okay I I did a small

6:47businesses if I run this uh this command it will migrate the managed uh the manageable if it's Sparky you can use this status which is create table btusd as and select star from high meta store and you select the table if I go now to

7:11crypto you see that's my my BTC uh USD UC has been created but of course you can also sync a table and that's what I did over here I'm creating a schema called futix then I would use this for cheeks and here you see I have an issue because tell me like you should have the same

7:35name of the table and yes so when you think your your upgrade your tables from high metastore to Unity cutter they should keep the same name so here I want to create Euro football while I should have added this Euro underscore football and again here I'm just okay uh here I just need to like upgrade and

7:59here I'm specifying sync table instead of sync schema same thing it says okay there is no issue over here so I can just remove the dry run and run this command and again if I go here I have food cheeks that has been created of course I have uh I have everything I need and now of course you can still go and

8:22query your hive uh metastore uh your high meta store table you can still do it but once you complete the migration if you want you can go and of course delete your uh your hive meters to your table and then you will keep only of course the unity catalog and now that you have created your

8:44tables you can go and for example Grant permissions to a specific for example I will get permissions to Ali azus was my colleague like I would write him like for example the select I would Grant the other permissions and from now you can benefit from the UC features while keeping your tables in the same location where they were just keep in mind it should be an ADLs gentle

9:09thank you for uh watching this video and see you soon