Build Governed Data Pipelines: Unity Catalog + Delta Live Tables

0:00hello my name is Yousef and today I'm happy to share with you a great news databricks has announced that public preview of unity catalog with DLT and for those who don't know what's DLT DLT makes it easy to build and manage reliable batch and streaming data pipelines that deliver High the quality

0:23data on the databricks lake house platform and DLT helps data engineering teams simplify ETL development and management with declarative pipeline development and on the upper hand Unity catalog is our governance solution for all data and AI assets in the lake house platform it brings a lot of feature like

0:45automated lineage for all your workloads it also has centralized governance for your data AI built-in data assertion and discovery and as I have mentioned databricks has announced this public review of DLT plus Unity catalog of course this is just a public preview so you may encounter some

1:07limitations that I advise you to have a look at but bear with me that those limitations will be removed once of course this feature will be announced ga but first let's go and have a look at the workflow so if you want to declare a DLT you need to go to Delta live tables and then click on create a pipeline so

1:31here you'll see that you can choose a pipeline name so we can call it for example football and of course you can bring your notebook and here you can specify of course whether you want to use high metastore or Unity catalog so of course once you choose Unity catalog you should have in advance created a catalog and then you can Define this

1:54schema where you want to push your your tables for my for for for my example I already developed two Python scripts Python scripts and SQL Scripts so for my case I have a data set that includes all the scores uh for I think the last 15 years in for

2:18specific click so here I'm just using uh like the integration of photo loader with Delta live table so I'm creating my bronze layer so where I'm gonna bring this data uh that I have stored over here then I'm creating another uh like table like storing the games for 2010 and of course I'm declaring the schema

2:41and then from this table I'm doing a union to get all the all the scores and then I'm doing some transformations to clean the data and then I'm creating my gold layer where I compute for example to get all the large scores higher than uh four four goals whether you're playing uh home or away then of course I'm getting all the home wins per year and of course

3:05away wins per year to get to get this so once everything is declared if I go to my uh to my workflow Delta light tables python you will see that it has generated my my uh my pipeline so we'll see the bronze like the game of 2000 the game of 2010 I did had this game uh game silver I did

3:31create new columns and then from this silver I created multiple uh multiple tables and if I go of course to settings you will see that I have created my catalog called football underscore Python and then everything is stored in uh in this database so if I go to data I will look for uh football python

3:58see of course I can still like uh handle the permissions so I can for example say uh only matcha and Yusuf can uh have like all really all the Privileges of course if I go to games then I will have the tables I have declared so I have of course my game 2010 so I have the uh my

4:20my specific schema I can also get the lineage again the same one I got there so I can get like the linear information of course I can also find from where the pipeline is coming so it shows me that that's the pipeline it's called you know that with the DLT you can also access the metrics like incinerated by this this pipeline

4:44so with this new version we have takes are much much easier so I did this example on databrick SQL using SQL uh serverless Warehouse so if I go to SQL editor now if I want to query the logs I can do it for example in my uh in the other uh using the Clusters but for my case I

5:07want to do it with DB SQL so you have a new function called event log you go called event log where we can add the DLT uh the pipeline ID and now that I have created this pipeline ID and I have created this table I can for example go over here I can get the lineage if I want

5:29I can for example get the runtime info I can get like the audits of the pipeline who did started and who did create the pipeline so you can get a lot of a lot of details I I will I can also show you like I would query this uh this table over here

5:54and I will use a serverless SQL Warehouse and I'll guess my data and of course you can uh then create dashboards over this specific uh metrics

6:16so thank you for uh attending this session