Monitor Data Quality, Drift & Model Performance Unified

0:00hello and welcome to this new video my name is Yousef and today I have a special guest with me this guest is aliasis and together we're gonna discuss about lake house monitoring highly hi everyone uh really happy to be here uh this tool is a game changer for keeping track of your data's quality and

0:23another performance uh this is an awesome to perfect Ali so could you tell me what exactly is the data breaks Lakehouse monitoring um yes uh for sure um so database take us monitoring enables teams to oversee all their data pipelines comprehensively this covers

0:47everything from data and features to mention any models all within a single tool eliminating the need for extra tools and intricate setups it's fueled by unity catalog so this features enables users to guarantee the Excellence the Precision and the dependability of the data and AI resources it

1:10um especially this by providing profound visibility into the lineage of both data and AI assets ensuring they meet the highest standards okay that sounds cool and why do you think is that important uh yeah good question so um in order to extract valuable Insight

1:34from your data it's essential to trust in the data's quality uh so by actively observing your data you can gather numerical metrics that enable you to Monitor and validate the consistency and Excellence of your data as time progresses um if shift in your data distribution or

1:59the performance of associated models become apparent the monitoring system provided by database Lakehouse can document and review about these changes uh aiding in the artification of their underlying reasons yes makes sense makes sense but how does it work behind the scenes yes

2:23yeah so um so Lakers monitoring on databricks involves setting up monitors uh linked to specific tables and this monitors enables the observation of table performance and which kind of monitoring is it yeah

2:45um so uh Dynamics latest monitorings um offers several analysis methods uh we have the time series analysis so this involves comparing data distribution within different time Windows uh we have the snapshot analysis so in contrast to time series uh snapshot analysis track changes in the complete contents of the

3:07table over time and finally we have the inference analysis so this pertains to tables holding predicted values generated by Machine learning classification or regression models so to summarize you select your table and the profile type so whether it's

3:29time series snapshots or inference and then what's next so after selecting the profile type and the chosen tables the latest monitoring will Define what type of metrics are really for the tables and it will create two data tables a drift table which contains statistics related to the data's drift over time and a profile

3:53Matrix table which contains summary statistics so those tables will feed a DB SQL dashboard that will be automatically built and used to inspect metrics with some visualization and we can also use those tables to create alerts multiplication to be notified when changes are detected or to set up a

4:15web hook to retrain a model if its performance is too low yeah it's very very cool but please now show me how it works uh yes uh for sure so to create a monitoring we can do it via the UI or via the API uh but uh like this we can just import package and create a user

4:39create monitor method um but here we will just do it via the UI so to do so we can open the data Explorer navigate to the table uh to the specific table we want to monitor so in our case it's Airbnb price Insurance logs so it's an inference table with inputs and predictions of a regression

5:02model and on this table we have simulated a direct on a few features for the purpose of this demo then we click on the quality tab uh on get started uh button to begin to begin creating a new monitor so it will be prompted to choose the profile for our monitor so we will choose influence

5:25then it's a problem type so in this case it's a regression problem prediction colon prediction eventually is a label column in our case it's price the metric granularities let's say it's every hour every day as Samsung colon which is timestamp and the model ID column

5:50uh we can also specify the schedule like uh once a day suited to the frequency of inference uh to track our model performance um then we can write the schemas are generated assets um uh like like the drift and provide tables so in our case we will put it in

6:12monitoring demo dot default the default schema optionally we can use the Baseline table to compare with previous values and we can also add some Metric glycine expressions and custom metrics uh really helpful if we want to track business metrics like uh churn rate or Delta of

6:36price compared to Baseline uh so um we'll just put the table name which is monitoring demo table default dot Airbnb at pricer

6:58Baseline Android has and then we can click on create so usually it takes some minutes but I read a few minutes ago for the demo let's click on this and uh so as a result uh once the monitoring is created we

7:21have uh so additional tables so drift Matrix tables and the profile Matrix tables and if I click on the table I am monitoring this one and I click on quality I have the monitoring settings and I can for example click on view dashboard which will redirect me on this

7:44page DB SQL okay but um it's it's amazing I mean to be honest I'm really really amazed but just uh for folks were watching this video it's very important to keep in mind that Lighthouse monitoring is still in public

8:07preview and requires some prerequisites like having Unity catalog enabled and acts having access to databrick SQL and of course the monitoring can only works for Delta tables and the compute behind scene is a serverless compute

8:29so besides those information uh Ali what are the interesting insights you can share with us yeah so on the SQL report uh which is really cool is that on that dashboard is that we have with that dashboard we have um direct insights on the most important metrics of our model with some

8:53interesting summaries statistics on our data so the report is uh splitted in sixth section so the first one is the stats overview so we can firstly get some statistics for the latest window like we can see that there were 284 inferences the last time and

9:16158 of these predictions are the 50 probability of being higher or lower than the actual outcome on top of that we have 100 Min absolute error and almost 200 root mean squared error and as we have added some drift in our data uh for the purpose of this demo of

9:40course and um as the monitoring does also even statistical tests for us like the coal mugurus mean of test uh the Drift We simulated has been identified as we have a case value of 0.65 which means there is a grain there is a great chance to have a few different distributions between the

10:02inference and the Baseline um obviously if we know precisely what we are looking for we can eventually also filter by time range or inspection window or eventually a specific model uh so this was for the first section then we have more information

10:27regarding uh the model performance uh with different metrics over time and if yeah if I go there um uh so uh for example in this graph which is really interesting is that we um that when we have multiple nodes we can see their performance over time on

10:50the same graph to compare them uh next we can get uh some insights on the data Integrity we can inspect the number for example of nine or zero values by colon um here it seems that uh that's a that's a quality of our data

11:13seems good and that's nice then we have additional information on the results of the statistical test you use to detect eventual data rate compared to the Baseline for each column over time and there is also an array to sum up the results for uh category categorical and

11:38numerical data and for example if we look at the column which change for its experiments accommodate which is a numeric feature let's say here dates oh yes it's here uh we can see that the

12:00value of the ks test is at some window equal to one which means there is a difference in the distribution of accommodates between the Baseline and the difference and the p-value just here uh the P value is almost zero it's zero which means that the SQL test

12:23is significant so knowing that there is a change in the distribution yeah it might be interesting uh to see summary statistics on this variable and this is why after the drift section uh there are additional details on the numeric and category called variables essentially a summary statistics like

12:47count of Rage media and um if we compare so the average of the accommodates variables in the input to Z1 in the Baseline we can clearly see that there's been a change in the variable between the first uh between the previous window and the last window

13:11uh so um knowing all these kinds of metrics is really helpful when developing a model and to monitor it uh especially when it's deployed in production as when addressed is detected we may need to investigate a deeper the course of these issues and eventually retrains and although using the new data or as a

13:35business impact of change in predictions that's it okay that's I mean I'm a bit lost uh I lost my words I mean this data monitoring experience makes three makes it really like makes this experience very very easy to Monitor and have this those incredible

13:58insights so I'm pretty sure like users will really enjoy using this uh this new feature again thank you Ali for uh joining this session and thank you thank you for inviting me today you're welcome mate and if you have any questions don't hesitate to share them in the

14:22comments below and of course remember to subscribe in order to stay up to date with our latest tutorials and see you soon take care