Unity Catalog: Centralized Governance & Data Lineage

0:02hello and welcome to this Unity catalog introduction my name is Yusef mourni and I'll be your host today let's get started so by the end of this session you will be able to describe key concepts related to Unity catalog manage users and groups

0:24create Unity catalog enabled cluster and SQL endpoints create and cover table and fuse govern file access limited table access with Dynamic views and the lineage that's before let's have a look at an overview of unity catalog and before we dive into the specs of unity

0:48catalog let's take a moment to discuss the more General topic of data governance data governance really refers to the process of managing the availability usability integrity and security of data data governance has become extremely important in today's data Driven Landscape and is needed to establish

1:10trustworthy data that is really accessible to those who need it in accordance with internally defined policies or externally regulatory requirements when we talk about data governance we can generally divide discussion into four main areas the first one is data access control and

1:32data engineers and platform admins need to be able to lock down data and data generating artifacts such as files sales models and Burmese access to only those who need it we have data access auditing and I mean of course any data governance program requires knowledge of how data is being

1:54used not just who has access to the data but when and how it's been used and the data I need and the journey of your uh of your data takes from its origin and as it moves through your pipelines it's referred to as a data lineage and being able to retrace it is very beneficial in a number of ways

2:18first it's much easier to identify root cause of processing issues enable you to predict the impact of proposed Upstream changes to your processing helps compliance team prove that the approved that data and reports come from like traced and verifiable sources in short

2:41it feels better understanding of the data for data analyst data scientists and data engineers and last but not least data discovery which is which is very very importantly because we need to locate data uh I mean when it comesification data it's very very important but the data of

3:04course a data governance program requires a readily accessible inventory of data assets and unity catalog allows you to Define access route once where they can be like applied across multiple workspaces clouds languages and use cases governing all your data Assets in a

3:27consistent way through easy to use uh interface and SQL apis the access controls available go far beyond what any cloud system can provide and while your data is ritual Unity catalog can manage it from all a central Hub in a consistent and familiar way there is no need to replicate or

3:51translate your security requirements across different systems if rule changes they only need to be applied once and unity catalog brings also like within the ability to do a fine grained audit of all queries performed to support data governance use cases and data lineage will also be collected and visualized

4:13for tables and columns across all languages and of course this feature is a ga right now and unity catalog is additive and works seamlessly with other meta Source including the Legacy High metastore built into each databricks workspace which of course many of you have taken advantage of and through the three level

4:37namespace we will cover shortly you can have seamless ongoing access to your existing data while it while makes it it makes it a lot sense to upgrade your data to Unity catalog as soon as you can and there is not no hard requirements upgrading data to Unity catalog is a simple operation that can be done

4:59whenever you like and today as I mentioned we're gonna discuss the key capabilities of UC and some of the broader pattern changes that you see is bringing as a service and Paradigm shifts bring to the database ecosystem so some of those will include like this centralized metadata and user

5:21management centralized it Access Control Data lineage data access auditing data search and Discovery and of course not now but secure data sharing with data sharing and how does unity catalog fit uh into the databricks landscape so here on the

5:43left we see an overview of how things looked prior to a Unity catalog and prior to originality catalog user management and data management were a function of the workspace and the workspace admins Define users and groups within a particular workspace either manually or through a connection to an identity provider or of some sort but

6:08the key Point here is that users and groups are ingrained into a single workspace with no interesting synchronization in multiple workspace environment and governance was provided by Cooperative efforts between a metastore and the compute resource accessing the data that is clusters and SQL endpoints

6:30The Meta store by default was a local high metaster although of course we did support external meta source and one key element to note here are the locality of the metastore which makes it challenging to easily consciously apply the same set of security controls across multiple workspaces

6:52furthermore the full full security requires cooperation between compute resource and the metastore if compute results are not appropriately configured to enforce access controls then access rules can be bypassed and by contrast Unity catalog with its enhanced security model sits on its own

7:15Factor the security related elements out of the workspace enabling the defined ones principle we discussed earlier in this architecture account administrator manage the users either manually or through a connection to an identity provider of some sort they also manage the creation of meta source which of

7:39each of which of course contain metadata and access control list users groups and metastore can then be assigned to workspaces as well and compute resources that can connect to UC will be by default be subject of to UC security constraints no specific

8:02configuration or Administration is needed to make the system secure and as can be seen security policies Define a meta store in a meta store are automatically and immediately propagated to all assigned workspaces I'm not that's now that uh the uh we understand some of the benefits uh UC

8:26brings uh as we all how it is fit uh in the rest of data breaks Let's uh take a look of some of the key element out of the unity catalog metastore so The Meta story is really the top level container of objects in unity catalog accounts admin can create and manage meta stores and

8:48assign them to databricks workspace to allow like workloads in those workspaces to access the meta store and the meta Source like The Meta Source store stores securable data assets as well as the permissions that cover access to those assets it is important to not confuse the UC metastore with the high

9:11metastore the high meta store which will be you are familiar with uh exists in the local meta store link to each databricks workspace and also provides data storage and access control capabilities we have course within the unity catalog meta stores we have one or more catalogs

9:33and a catalog can contains a database which also refer to as a schemas that are connected to UC also present of course in this catalog we have the Legacy High metastore under the name Hive underscore metastore and deeper like diving deeper in the objects for your ER Key we have the

9:55database also as I mentioned referred to as a schema the this concept would be familiar to most of the concepts are unchanged in UC then we have a view at the very bottom of the hierarchy within the database we find our data assets like bearing assets table and fuse the concept of the view is unchanged it can

10:17be taught as a really like uh read only a virtual table that is executed when you query it and of course as we uh mentioned databricks support two table variation manage and external tables so let's discuss a bit so we have the manage and

10:41the manage tables are the one that are Store where the data is stored in the uh your ADLs or your SV bucket which means a databricks manage the location and what we have added recently is the

11:03possibility to Define uh the location of this manage table the only thing you should keep in mind that manage table supports only Delta tails this is very very important and we recommend you to use manageables because you don't have to Define permissions like storage credential and external

11:26location but in the other side you can also use the external tables and the external tables are also like tables where we're going to Define of course the location which means this can be another uh another data Lake of course you need to Define storage credential and external location and that will give you some

11:49details about it and in contrast external table supports like several formats like CSV parquet and other uh and other things and there is also a key difference between those two is that in external table if you drop the table you only drop the registration in the metastore and the data will still reside

12:11in the external table and I will do a comparison of those two tables in within few minutes and then we have something very very important called storage credential and you need to define the storage cut also whenever you have an external table and the source code also encapsulates an authentication method to access the cloud storage and we have also the external location

12:35that it also need to be defined and this external location combines a storage credential that you have defined previously with a cloud storage path providing an object that it fully describes an external location including a way to access it and this is very very important and this is just a comparison between those two tables so the method

12:57the for both tables limited data lives in the control plane for the manage table like uh supports Delta only while for the external supports several formats it's dropping data in manageable discards data while for external drops leave the data intact and of course like data lives in metastore managed by

13:20storage location for manageable and data lives in a user provided storage location for the external table and the quest like when to use external table so it's quick and easy like from external table in Hub meta store when we have like external readers or writer when it requires you know a specific storage naming or hierarchy when you

13:44need this infrastructure level isolation requirements and of course when you uh can't have a Delta uh support requirements and regarding the trail level namespace so for as I mentioned prior to UC a workspace was effectively a logical point of data segregation which you see

14:06we need to add an additional layer of data segregation so we added these third level names namespace called a catalog so a catalog as I have mentioned previously is an object that contained databases like uh database like you are familiar from from hive and traditional error dbms contained views and and Tails

14:33and we have also like data access control which is gonna let you you know graph a print access you can do it either through the API or through the UI or through terraform so you can you can grant permissions over a catalog over external location create share schema I mean you can Define every uh permission and as I mentioned you can

14:57either do it through the API uh through the SQL through the UI as you can see in the screenshot or using like ANSI and this is NC SQL line this is just an example like Grant select on I don't know event for example to Engineers which can be a group or to a specific User it's really up to you to Define how you can do it

15:20and regarding uh like the new compute resources so we have the first type which is a single user and other name suggests cluster using this mode can only be used by a single user who is designated when creating or editing the cluster others of course cannot attach this cluster to their notebook so this

15:44app this apparent design is encountered by its functionality that make single user cluster supports SQL python Scala and air are workloads and we have the second type of cluster which is shared uh shared and it's this the shared cluster ports can be used by multiple users

16:07and for now it supports only SQL and uh python and tables in in UC so as I did mention before we had this traditional two layer namespace now we have these two three level namespace so before it was schema dot table now we have the catalog dot

16:29schema dot table and this is just I'm gonna just a brief summary of uh how table access control works in unity catalog so let's suppose you wanna access to what give someone a permission or a user permission over a table you need to be uh to Grant him the usage over a catalog the usage of our

16:52database and then deselect over disable if you want him for example to update this table then you need to Grant him the modify over this table if you want him to create another table you need to Grant him the create over the database and if you want him to create a database we need to Grant him the create over the uh the catalog

17:15and of course just a brief uh summary we have two types of tables as I have previously mentioned so we have the catalog when it's managed to just you just need to define the manageable when it's an external then you need to Define what we call the external location and the storage credential

17:37and as I have mentioned like this restaurant also just when you like print like it's to connect to enabled UC to connect to this XML cloud storage for as I mentioned it's equivalent of uh IM rule for example for S3 and for the external location it's just the storage credential plus the uh the cloud path and of course over also those external

18:01locations and storage credentials you can Define permissions like create table read files write files now we have views and Views are very very important in unity catalog so just keep in mind uh when you're creating a fuse and you're gonna do like data masking or anything else make sure to use shared clusters because if new

18:25shared clusters so let's suppose you want to create a few so you just need to have the select over to access the view you need to have the select of course over this specific View and to create a view you need to have the possibility to create uh to create over the database and we have multiple

18:48uh uh multiple options with the views so we can do like limited access to columns or limit access to rows or also do data masking which is very very important and for data lineage which is I think one of the most feature most interesting features also in unity catalog so the

19:10lineages as I mentioned pretty exciting so you have the ability to automatically capture runtime lineage on clusters and SQL endpoints down to the column level so that means that if you create like derivative data sets based on another data set through the spark apis you will see this relationship uh in the ndui

19:34and so how does it work so we have this service lineage so which captures either like tables created through like either your ETL job or ad hoc and this is captured and then everything is written there and you can explore this uh UI in uh the lineage in the uh in the UI and like and how it goes so so before spouse

19:58goes goes Ops and paralyze this operation so to its work nodes it's creates something called a logical plan and this is let's say an abstraction of all transformations in link and craft form and the metadata of these objects gets submitted to a lineage service this runs of course in the control plane and this service assembles all the logical plans for all the commands together to

20:22create the linear lineage graphs that can be boiled down into this tabular information about what tables fed into another tables and what columns fed into into another one and now let's move to the uh to the demo so just to show you as I mentioned the compute so you can have single user and

20:47shared so single user as you can see you have this Unity catalog tag same thing for the shared if you choose like no isolation you see that there is no UC UC tag so let me just go to Unity catalog I'm just going to see in case okay I think the cluster is on so the

21:10first thing we're gonna do I'm gonna create a catalog called demo underscore UC so I'm going to create the catalog and I'm gonna use this catalog I'm gonna also create something like a database called a bow because I'm going to use Titanic data say we can also use this boat and then the first thing I'm gonna do let's suppose I have a data set in the

21:33Legacy High method server which is high Met I underscore meter third defaults per K so to migrate my data for example to Unity catalog I can just run this this park creatable demo unders demo dot to this demo you see both I'm gonna call the table Titanic underscore V1 and I'm gonna fill it with this data coming from

21:55this specific table and once this table is created I'm just gonna run this describe extended and you will see that I'm gonna create my first managed table with the unity catalog and of course the format as I mentioned manageable supports only Delta tables so this is the information

22:18[Music] um and you see a map type manage and of course provider Delta if somehow I want to create a parquet table within my manage tables you see I would have never told me only Delta is supported for manage tables and if I have for example a Delta table in my high meta store I can also

22:41do run a clone instead of like running create such as I can use a clone and here you see I'm going to create this Titanic V2 deep clone and of course within you see whenever you create a table there no permission is granted so what I can do I can either like Grant permissions through the SQL API and here

23:04we have something called the uh like here I'm gonna Grant all privileges like uh on catalog and there is what you call now inheritance which means if you grant permissions to a catalog it goes through the through like the database and table or you can grant permission on each layer so for my case I'm going to range all

23:27permissions so now I can show you the permission over the catalog so show grants you see all privileges we're gonna show grants on the schema again and so so Grant on uh like a specific specific table and if I go to the data Explorer I can click on data and I will select of

23:52course my catalog which is demo UC you see both I can when I go to demo you see I can grant over here the permission you see I can grant like custom permission to a specific user or group I can select either all privileges or a specific one so it's up to you to decide whether we're gonna Grant produce permissions within the UI or within the uh within

24:16the API and now if I once I can now query like say my first my first step and when it comes to this is when it comes to manageables when I wanna like create external uh location external tables I need to Define first this storage credentials so I I need to create this storage credential so you

24:38see here recommended is manage identity so I need to specify the search credential name the access connector ID and I also need to specify the external location so I can specify the URL specified search patterns I have created and some comments if I want to help for example achieve and for my case I'm gonna use this uh X this uh external I'm

25:04gonna show all the external locations I have over here so for my for my case I'm gonna use this one but what I need to do first I need to Branch myself the uh some some permissions over uh the over the external location to be able to I'm gonna reference my self determination over this or upper Villages and I'm

25:28gonna just Square UK so now what I'm gonna do I'm gonna just uh my words [Music] um I'm gonna describe the first external

25:51and I'm gonna again show all the storage credentials I'm gonna use of course this one you see Excel you see they can have details about who created and when and now uh what I'm gonna do I'm just gonna show Brands over this uh this external location so we see I already have the access and I'm gonna also show

26:14grants over this Suresh Cadence and I need to change it and now I [Music] um okay there is always a bug or stock okay

26:38this does not exist ah okay let's do some table again so there is a storage we don't feel I have

27:01mentioned okay ah my I was I was copy pasting on the wrong uh on the wrong one okay

27:23my bad I got lost uh it's it's this okay I was doing the wrong uh the wrong thing so now what I'm gonna do so let's this is of course demo H live demo issues uh uh let's now I'm gonna create the my

27:46first uh external table I'm gonna call it Titanic external I'm gonna specify the location and then I'm gonna say Okay select count passenger ID so when it's survived equal to zero then that else uh survive so I'm going to create a new column called status and I'm gonna compute this first

28:08external table and I'm gonna do this describe uh external okay so is this you see type external of course this is Delta and if I want to create like the same another table but this one is external but this time is parquet I can also create a parquet but

28:30before you see I need to drop because if you're using the external table like the uh all the other formats reports only appending data if you want to or The overwrite Works only if you are using a Delta table so now I just needed to drop the data there

28:52and now you see describe extended I can have access to my parquet uh parquet table s and now that I have created this table so you see that I have some specific uh let's say a relationship between those two tables because I created this Titanic uh sorry this Titanic external based on this Titanic V2 so what I'm

29:16gonna do just gonna go to data I'm gonna go to demo you see both I have Titanic external so here I can click on lineage if I click on lineage so you see that I have a I have created this uh

29:38uh Titanic V2 based on Titanic I create Titanic external based on this Titanic V2 and you see this is very very important you can also see that I created another one which is Titanic external external packet and that's not everything so you see I created the column stages which is whether this past this

30:00passenger is dead or alive so you see status it shows that has been computed from the column survived so this is really how how this lineage is very important you can also see whether this whether this uh where this table is called in which notebook if this table is on a specific workflow

30:22whether upstream or Downstream and if this table is called on a dashboard and this is this is super super uh super super important and you can also like see frequent uh frequent queries you see all the different types over here and again we can uh we can grant uh the permission and also uh I sometimes I

30:44mentioned uh I didn't show you so you can also click on over here and you can see like whether there is lineage or not so this is super super important you can also export this lineage with uh with the API and of course that's not everything so let's suppose now do you wanna uh uh you want to Grant some I'm gonna

31:07create some some views so we're gonna first of all try this column level permissions so over here I'm gonna create a view called Titanic red exit and I'm gonna specify like select the number and then say okay if I am part of this group then I can see the gender else if I'm not part of this group which is uh me a then uh I will see only

31:30dashes and that's it so I'm gonna run this square right now and because I'm not part of this group EMA so you see I want to be able to view this specific column this is when it comes to column level permissions for row level permission so I'm gonna create this Titanic redacted Pro I'm gonna select like data from this

31:53specific table and then I'm going to specify condition if I'm part of this group I can see everything else I will only view where the number is lower than 5. so I'm gonna run this view you'll see I will have only one row and I'm gonna create again the same view but this time I'm gonna mention a group uh where I

32:15exist so I'm gonna with the same same query only change the group so if I run the second view you'll see I can view everything so this is how we can do uh uh column row level uh masking and something also which is very very uh very very important with Delta tables

32:38now you have the possibility to create what we call primary keys and foreign Keys just keep in mind that this primary keys and phone case what enforce Integrity but they are declarative they will help you to create your data model within for example Tableau or your power bi or power bi or any other connector supports uh Unity catalog and also helps

33:02like the data analyst or data engineer to understand the relationship between tables so here I'm going to create for example table persons it's kind of constraints person's PK so I'm gonna specify primary key those two two primary keys and I'm gonna also creates like table beds the constraints and I'm gonna specify the foreign key so now that I have created those two tables

33:24if I go to data demo UC I go to Features now if I click for example on Persons so we see primary key primary key if I click on pets it shows like primary keych references table uh persons and this is very very important and you have also something you can leverage super uh

33:47as you can leverage called information schema and this is where we can find everything we have defined in your within Unity catalog like contract constraints referral reference everything like super super interesting and you can also Grant permissions if you want over here if you go to like the metastore permissions without having to access like the uh without having to

34:10access what we call the uh uh accounts console you can for example grants permission to someone to for example to create a catalog track to create an external location to create like a provider and everything so UC is very very important it has it has also a super great integration with with SQL so

34:34whenever you create for example a warehouse a SQL Warehouse you will see that you can activate enable UC or Not by default you see uh is enabled and you can access the same and you can own something important you can only have one metastore per region and one metastore assigned uh to a specific

34:57workspace so thank you for attending this session and see you soon