Master Unity Catalog: Step-by-Step Tutorial

0:01hello my name is and today I am introducing a very interesting feature called peterstore I'm pretty sure you all know this feature but now we have an integration with unity catalog and to get started you will need a database runtime 13.2 for machine learning or a standard runtime back if

0:25you are used in a standard runtime make sure to use the pp install in order to install the databricks feature store Library for this specific example we will use the New York Yellow Taxi first we will start by uh we will start by having or by using this

0:49specific diagram so we'll have our raw data and from this row data I will create two filter table Feature Feature store table when I create the pickup theater and the drop-off features and then those two tables will be used for to join it with the raw data to train a model then

1:11we can register this model and we'll do the serving of course if you can uh choose this data sets it's on dbfs just look for the data breaks underscore data sets and look for near your taxi with zip code and I will uh display uh display the data

1:36and you will see how the data looks like and then in the next step what we will do as I mentioned will will compute two tables the first one will be the pickup features and the second one will be the drop-off features of course here we're gonna compute two new columns or two new features the

1:59first one will be the countries so we'll use a window of one hour and we're gonna slide like 15 minutes and we can also compute them the main fair amount of course by having a window for an hour and a sliding of 15 minutes that would be the same window and for the drop-off features we're gonna count a number of trips within a window of 30 minutes and

2:23whether the thrifts end on a weekend or not so let's go back to the table so we have the uh pick up daytime the drop-off uh date time the trip distance the fair amount pickup zip and the drop off dip so the first thing we're gonna I'm gonna create to helper function so the first

2:46one uh the first function will give it will give as an input uh the daytime and it will tell me whether it's a weekend or not and for the second one there will be a filter so I'm gonna give a data frame gonna specify the timestamp and then I will specify the start date and the end date this is just to filter the data frame to get a

3:11specific timestamp and then in the next step so I'm gonna Define two functions that will be the function to create a table called the pickup feature and the second one will be the drop of feature so for the pickup feature as I mentioned we're gonna call the function to filter and get specific or subset of my data and then I'm gonna

3:34call this the pickup zip I'm gonna I will have a window of one hour and slide in by 15 minutes and this will help me to get the fair amount and then the computer to compute the number of uh pick up of course I will sell I will get at the end the pickup seat the timestamp

3:57the main fair and the countries the count of trips that's for the pickup feature for the drop-off feature that would be the same thing this time we're gonna get the number of tricks for to the for a window of 30 minutes and whether it was uh at the end of tripod weekend or not and now what we now that we have those two functions I'm gonna uh uh get my two

4:22my two data frames so the first one I will give it as data from the raw data the time step column it's the pickup daytime and I will I will give uh start and end date to get the data for only one month for J for January data and same things

4:44apply for the drop-off feature the same row data but in different timestamps and of course you get the data for January and let me show you now uh how the data looks like so I will have uh this deep the timestamp the main fair you know window for one hour and of course the count of trips so

5:08of course I could have displayed the drop-off feature as well and now since it's backed by unity catalog I'm gonna create my catalog and then I will create my schema so the catalog is called ml underscore feature underscore store and then I would create a schema called a taxi underscore example

5:30I will call my feature store client and then I'm gonna create my first my two first feature store table by calling the fs.create table so I'm calling my catalog my schema and the name of my table of course I would need to have the primary keys so how I'm gonna join this

5:53table with the row data I will have the timestamp uh the timestamp keys and of course the data frame I will use to fill the specific table and of course the description and same thing will apply to the to the second table that will be the trip drop off time series

6:15feature so those those five uh parameters are very very important and of course this is the first step uh to create those feature store table and later what you can do you can uh

6:37add the data using what we call the right table of course this can be over here this can be the case you can have a job that each each month it will compute the new features and then they will push those new features to the to the two features table using the right underscore table and of course you will

6:59need to choose which table you're gonna push this new data so again my same catalog same catalog schema the name of the table the new data I will input and then you have the mode merge of course you have two options you can either merge which means you're going to append the new data or you can override just overwrite everything and this step

7:22can be taught as a job that will trigger its month or each day depending of uh on the SLA you want to have and over here you see I'm getting the February's data so in the first for the first first time I computed generate data and now I'm adding February February data as well

7:45and of course those tables you can query them as you want since they are Delta table you can grant permissions and let me uh let me walk you through this through this data so it's called feature ml underscore Feature Feature store so let me show you the data

8:11it's ml teacher store tax example and you see we have the two tables first you will see the primary key you can see it over here you can see the history the this to the the description and of course you can also

8:34grants the uh granted the permission over here you can see to whom we're gonna Grant permission whether it's a user or a group and which kind of permissions and at the same time I will go to now to feature store to look for for my specific sites called ml teacher

8:57all right I should have checked over here so it's ml feature still so you see over here I have my two feature uh theater table so let's click and check you can see over here when it was created who created this data the data source the last time it was updated uh managed the primary key you see the

9:21zip and you see over here the uh the timestamp keys you will see the producer whether there is a job so as I mentioned you can think about having a job that will update this data in multi-base or weekly base and of course you can see over here the two uh the two filters I have mentioned

9:47and now what I will do now that I have my filters let's Now train the model so this function will just get me the latest model uh model version and now what I will do I will call my two filter table and I will do a feature uh I will do a feature a feature lookup

10:09and this this step is uh is very very important why because this feature lookup is a function that will allow you to retrieve the values of the feature from an entity and of course it uses a feature table to find the latest value of each feature at of course at a given date and we have also something very very important over here so you see I

10:31will write the name of the table the feature which is which features I want to take the lookup key to join and the timestamp lookup key and the typeset key is a property that you can set on a filter table to enable the point in time correctness so this uh this point if time correctness is to enable this you

10:55need to configure your fit your database feature store to include a timestamp column in all the features table and this timestamp column is used to track the time at which the value of each feature is valid and this is very very important so we'll do this twice for the pickup time and the drop-off uh drop-off time and

11:18after joining so I will just end in case I have some uh pipelines running and I will start to run of course I will exclude those uh the pickup the pickup uh date time and the drop-off date time and I'm gonna train uh call the function create training uh create

11:41training set which is also a very very important functions in uh when it comes to feature store so if this sections allow you to create a training set for machine learning models using the filter stored in in future store and of course it takes three uh parameters so the list of the table the start time and you can also optionally add the SQL feature

12:03that's gonna be uh that's gonna be applied and then I'm gonna of course train uh uh get this tree instead loaded and let me show you how uh the data looks like obviously the trimmed instance uh the pickup the drop-off zip the main Fair win the one hour the country window one

12:26hour to countries drop off his weekend and the fair amount and the next step I would go and train uh train my model so you see over here I'm getting the columns I'm transforming this the data frame to our pandas this spark data frame to open this data frame um doing generation the test uh entrance

12:49and training we're gonna enable the autolog and then we're gonna train uh we're gonna train the model and after sharing the model we need to log the model on a registered model name so for my case I'm specifying this mod registry will be called tactic example pair underscore Unity catalog I will log the model and after logging

13:13the model I can now uh core or start scoring in a batch uh batch inference and of course this data will show the uh the fair amount so that's the fair amount that trip distance pick up the drop of the drop of drop of C and the next step what I will do I will

13:35now do or run the prediction using the latest in case for example I have multiple models I will I will use the latest model to run uh the predictions so here I'm using the mod URI and of course uh the data frame that I'm gonna use to run uh to run the prediction so now I can just go and view the taxi fare

13:58predictions and get the uh the results and I can go back over here I'm going to click on feature store again call the ml ml peterstore

14:21and you see I have this model which is pushed to taxi underscore example Fair underscore uh Unity catalog and I have this small one so we can see everything through this feature store uh to this feature storage UI and over here you can see the examples over here the fair amount

14:45predicted was 73 and the actual amount was 61 uh same thing for this one's 67 I got 73 and and so on so that was everything regarding uh feature store thank you for for thank you for watching this video