Unity Catalog OSS: Open-Source Data Governance

0:00a catalog does a couple different things now we're seeing these like AI agents like people are building these agents that can go call tools llms they aren't very good at math um so it's actually easier to have them write a little bit of code that does the math to open up um the manage table create and write uh apis which have been um long requested and I know many folks are excited about this that the resource Hier is this sort

0:22of like three level name space where there's like catalogs at the top and then schemas and then schemas can contain objects hey team today we're back with Michelle Leon and Quinton as usual for those who missed the first episode Michelle Leon is a product manager in charge of unity

0:44catalog Delta Lake and Apache Iceberg and she's back again to talk about Unity catalog open source hi Michelle hi thanks for having me back uh great to Great to chat again I'm excited to talk what about you Quinton yeah so um so Michel why don't we start by in know re

1:08discussing a little bit about Unity catalog uh what's the open source version like what is it doing why should customer you know or not customer like what should user uh care about it and have a look yes yes okay great so yeah I'll start off with with an with an overview of just the project the Genesis and just quickly recap that and then um I'm really excited cuz I'm going to show share more today about um and I

1:31also have some demos prepared too for the stuff that we're doing with um the latest 0.2 release yes I was just I was just working on the demos just before this so hopefully all goes well um but the uh just like a quick overview of the unity catalog project um it's something that was uh born here at Databricks around four years ago we started working on it inh house um in a lot of ways so I

1:53guess I'll quickly recap what a catalog is a catalog does a couple different things it's both like uh some people call it like a technical catalog where it does like metadata management about all of the objects um or tables uh and files that you may have in object storage and stuff like that and it takes care of the metadata things like okay this set of files this directory is actually representative of a table with

2:15this name and this schema um and you know also like maybe collecting stats and stuff like that um and then uh catalogs have also really evolved to take on like a greater and greater part in terms of both um like say commit coordination right so like Iceberg and then no Delta also is like moving towards like catalog based commits um and then also taking on a greater and greater role in terms of governance so

2:39the next level Beyond metadata management is how do you think about you know catalog being that source of Truth for Access Control and things like lineage and auditing and discovery of all these objects um and different types of tables or and you need catalog it's multimodal it's like volumes like unstructured data or AI assets and ml models that that all might be um in your

3:02data and AI estate okay so that's what a catalog is and un need a catalog uh four years ago we started working on it here at Databricks um to really like we were working with Hive metastore before so Hive metastore is kind of like the like leading open catalog standard right now but it's showing a little bit of its age um because it's been around for quite a while and four years ago we started working on uh Unity catalog to

3:25address some of the limitations that we were seeing with with the hive medic standard which is uh limitations around like governance and being able to do things like like credential vending uh which I'll talk about in a little bit um and also limitations around being able to govern things Beyond uh tabular assets so being able to uh govern yeah

3:48like you know unstructured data like you know your PDFs or images and like all of these like tons and tons and tons of like unstructured data that organizations are collecting now right and then what are they doing with that they're training models um they're like deploying like applications and and um uh all of these like different use cases that are like really exploding um over the past uh several years and so Unity

4:12catalog we've been building that we have um over 10,000 customers running it in production now um and data breaks really successfully and then just uh just this past summer actually in June is when we um went live with open sourcing Unity catalog so it was always really our intent to open Source Unity catalog but you know it took us some time to like feel like we hey we feel like we got something right here and then we really

4:35saw this like Gap around this like having this open standard for a data and AI uh catalog um that also supports like multiple different table formats and so um we went live with that the 0.1 release uh over the summer and then just recently um a few weeks ago back in September we went live with the 0.2 release which is a chalk full of updates

5:00and uh yeah I'm I'm excited to dive in and Yousef Quinton you know lead me where you want to go you know um when I I think I think he went live I think a few weeks ago and for some reason I was on LinkedIn very late because I'm based in Europe and um I saw that you see uh like v02 is live and I like okay let me

5:24check the like the release notes and I start okay identity authentication ml mode like UI uh like support for spark and Delta I was I I was super excited and my wife told me listen it's um 11:00 a.m. 11: p.m. let's forget about data breaks let us watch our my [Laughter]

5:47movie that is so that is so funny so that was the little anecdote here and so please Michelle can you tell us like I've seen like this huge release like ju like uh a lot of exciting features like one of them was about identity authentication authorization can you tell us more about this part please yeah absolutely so um something

6:13that that we uh added so we added like basically a ton of ton of different things and actually you gave a great overview so I guess I'll quickly cover like and I actually have some slides too that I can pull up um but but we added like a ton of different um functionality for governance in the .2 release so in the 0.1 release we started with some of these like API Primitives um like the core set of apis for

6:36metadata management for different types of objects like tables and volumes and functions um and you know with the UC server and the CLI and all that stuff and then in the 0.2 release a big core focus of us was to bring those governance Primitives um so identity authentication the concept of users right and then users that can have different privileges um to be able to

6:58access or or write to different um objects like tables um or volumes and stuff like that um in the catalog and then that is built on uh another like building uh block which is credential vending uh which is something that we um had in the 0.1 release and then we've kind of uh beefed up and added uh support for all three clouds for in the 0.2 release so I have a couple slides uh

7:21I'm going to share my screen let me see I didn't quite test this so hopefully it all works um all right hopefully folks can can see it see everything all right yeah amazing so I have a few slides here um I'll skip I'll skip around these um just uh just to kind of just pull these up as as

7:44reference while we're while we're going through it but here's some of the stuff that we added around identity um authentication authorization um so at a at a high level um we support uh authentication with external idps so the Google IDP uh with ooth 2 um token exchange uh with and we have skim 2 support for user management so we're

8:06adding this concept of users you can authenticate um and then also you can authorize those users to do things and so we have object level grants uh similar to SQL grants um I'll send up I'll throw up links for like the release notes and documentation and stuff like that right I don't want to just like do a super dry technical Deep dive of all these different privileges but at a high

8:28level we've laid the core Foundation of permission Management in the 0.2 um starting with these like course grin essential privileges at the catalog level um around like you know you use catalog uh schema um and then like select uh read volume for example um and execute uh for for different types of functions and so we these are like the

8:51core building blocks and then uh that are available on the 0.2 release and then we plan to of course extend these to other privileges in the future but as you can see there is already a ton of work that kind of went into even just this this like okay so now we have users and you can have you know they can be authenticated and we have like you know Google IDP support and oo support um and then uh also all these object

9:14level grants so that's a little bit um of a high level of how some of the like governance Primitives that we added I do want to touch on and then you know stop me I guess if this is if this is going to in the weeds but I did want to touch on something that I mentioned ear which is credential vending so you may or may not be asking yourself uh okay

9:36great so I've authenticated and I've granted privileges um to use or read from different objects right but how is that actually enforced and so how this is actually enforced is with this mechanism called credential vending that Unity catalog does so um essentially Unity catalog you connect that to object storage say like um

10:00uh S3 or ADLs or um GCS and then um what you do is when the engine is talking to the catalog to two Unity catalog it'll actually ask for so first it'll authenticate right with that with that token that we talked about you needing setup in the previous um in the previous uh slide it'll you say okay

10:24here's this token you know mapped this token is mapped to this um princi right that has these privileges uh maybe it can read from a certain table or something like that and then it'll ask the uh the engine will ask the catalog and say okay hey here's this token I'm trying to read from let's call

10:46it like say my table or something like that in S3 and so then Unity catalog will first authenticate um the user and also ensure that that user is authorized or that that principle is authorized to access and read from my table and then if it is then it'll actually vend out a a Tim limited Downs scoped storage credential um to access only the set of

11:11files that is needed to read from uh in this case my table and so this is very powerful and I know I'm spending some time on it uh but I did because this is something that like you know probably folks may have heard in other forums and um and it's very important to emphasize that this is actually the way that um that governance and this authorization

11:34is actually enforced and so uh because of course the server can actually refuse to vend that credential right and so what this does is ensures the engine and the user only has access to the files that are needed for that table doesn't have access to all of the files that might be in object storage which they shouldn't have access to um and then this mechanism is also the foundational building block for things like uh grants

11:56right which we just talked about and also like uh roles based Access Control attribute based access control um and a bunch of other different things um and then uh I'll say this also applies to any object that might be in object storage uh or any collection of files and tables volumes and models all share the same uh uh property of essentially just being a directory in object storage

12:20and so uh we do credential vending for all three of these objects so unified governance across data and AI okay I said a lot you're probably you hopefully uh hopefully um uh hopefully we're able to throw up these slides and stuff like that too and some that's amazing so wait let me just do a recap so um as

12:42let's say and I mean I would go to the UC UI which I think we'll talk just after and I can say Okay Quentin has access to for example this table of the folder within the catalog UI then I plug uh let's say spark or Doug DB to the catalog it's going to authentification me with my quincin email for example and then when I request access to the table

13:05like UC is going to say okay he has access and give back uh like temporary link to be able to access all the data and then NDB give me the answer that is that correct yes essentially yes and then um you know if you don't have access then it won't give you uh access to which is I think also the important part um

13:28it's always funny to demo that though because it just looks like demoing and an error um but it is exciting uh to also you know it's really important for for for governance um for for modern organizations and and you mentioned something very important that you see open source is a data and AI catalog which means we we handle the governance for tables but also ml models

13:53so I think and this is part of the uh V2 v0.2 release where we can I think handles the governance for the ml model that are registered in in unity catalog but technically how does it work like you you train a modeling data brakes and you register it and then you grant the access with the UC OSS open source UC

14:17OSS version yeah so here what I can do is I actually probably have a demo which I think just demos like pulling all of this together I have a few slides which I'll just quickly I can also just quick throw these up but it's maybe even just explained with the demo so I'll just I'll throw up the slides first so uh something that we added really excited about this this is actually probably the thing that I'm most excited about

14:40besides the UI and the 0.2 release which is uh adding ml models um as uh governable objects to Unity catalog um and so a little bit about like models cuz I mentioned and maybe this I mentioned that hey models they share this property of essentially just being a directory of files and object storage and so just like to double click into that um

15:04another uh project so you need to catalog by the way it's in Linux Foundation um Ai and data Foundation uh and we have uh another project um that data breakes helps to maintain in the Linux foundation and foundation called ml flow and so we actually added an integration with ML flow um in this 0.2 release as well and MLF flow has basically this this way of like packaging up model artifacts that can

15:28then be deployed on all these different um uh services like on Azure mL of course it can be deployed um on Databricks uh kubernetes uh you name it uh you can also run it as a userdefined function from spark um and so mlflow has a standard way of packaging at model artifacts which on storage is essentially just a directory of uh these

15:50uh files with with various metadata and so um mlflow also has this concept of kind of like a model registry where you're able to track like training runs and versions of models and you know we thought hey it's super important actually you know a a user might be running a piece of code which needs to go access different data sets like tables or not unstructured data that's

16:12in unity catalog they also might be needing to like train a model right it's the same user that's doing all these things and they want to have unified governance also for all those things because it doesn't make sense to have duplicated um access policy policies across different systems and so um MLF flow uh and the unity catalog um basically Unity catalog can act also as a model registry so we have this

16:34concept of like a registered uh model and model versions um in unity catalog and then we provide governance um on those uh and so a little bit I'll just flash this up and actually I'll just demo this this is it's way better as a demo but um we also have like an ml flow integration um which uh I can show in a second but first want to make sure that

16:56I was I was answered any uh questions as able cover this well yeah I love it so um I love the fact that we use M flow because M flow is like it's some kind of abstraction layer on top of all the models that makes everything super simple and has been using like it's such a successful project with so many you know user um I

17:18think it's a really good match yes oh so excited to collaborate and see all these things um all these things come togethers project which is something important yeah yeah absolutely I think with unity Catal I mean like you know open sources in Databricks is like you know history right like this whole concept of like the open open Lake housee it's like it's like our heritage it's like the you have

17:41open source engine um you know open source catalog and then also the open source storage formats um all of these things are are to give customers and users like maximum uh flexibility and control and the ability to like Leverage the best tool that works for them right yeah exactly so I guess you have a demo for the ml

18:04flow if I'm not wrong yes yes all right get get excited let me show you let me show you this let me show you this okay this is I'm a little nervous actually because I just did this yesterday though so hopefully it it should all should all be good all right okay so um I'll actually show a couple I'll show a couple things first

18:27of all can you just please let me know if you're able to see my screen you should be seeing the unity catalog UI yeah UI yes amazing here I'll zoom in a little bit too yeah yeah okay um so this is the uh Unity catalog UI uh so you can see here you can browse um all the different cataloges I have two different catalogs uh Unity uh default uh Unity

18:51catalog is like the default catalog that comes out of the box when you just follow the quick start guide um and then uh Unity catalog has overall like I should just say it's like there's a the resource hierarchy is this sort of like three level name space where there's like cataloges at the top and then schemas and then schemas can contain objects and objects can be um like I said because Unity catalog is this multimodal catalog uh can be uh tabular

19:14data to tables it can also be unstructured data in volumes right like maybe you have some Json like some semi-structured data like Json uh or you have uh just like a whole bunch of like images or PDFs uh right so here's the tables um also so functions um so functions uh these might be like uh SQL or python functions right that you might

19:36be um uh leveraging as udfs or leveraging from uh gen tools and agents which you know I'll talk more about that um as I talk up a little bit about like what's coming next in the 0.3 release um and then uh models so you can see I don't have any models today no registered models found so that's that

19:58uh that abstraction that I was mentioning earlier so let's see if I can go uh create a model in my catalog and also get the uh the do a first training run of my version so I also guess I'll show really quickly actually flash up on the screen the ml flow UI so as you can see basically I'm running Unity catalog I'm running um MLF flow um

20:20I have a UC server up and running right now and then what I'm going to do is I'm going to hop over into uh this tab I already have this demo script that's setup what it's doing is basically just uh connecting configuring uh ml flow configuring um connecting to Unity catalog as the model registry and then um uh I'm going to train like a small

20:42toy toy model so let's uh get this working so while that's running I totally didn't test this oh look okay I think I'm pretty sure this actually worked so great uh what I should expect to see is that we have a new model created here um in the unity. default uh the schema and you can see here it has I

21:05have this new registered model I can add a description for it here um you can see I can I just created it it's uh the morning here in San Francisco and then also you can see I have this first um uh version this first training run and so uh you can also see that the training run you can see it appear here 32 seconds to go in um mlflow and so then uh maybe say I like made some changes to

21:29uh the other training data sets that I have uh governed here in unity catalog and I want to actually do like another uh run um or experiment to see how it works so I can just do this again and then what I'll expect to see is that there's another version too of the model that's registered here so um super powerful to be able to do all this together right with also your like

21:52tabular data sets your volumes the data that you're using to and then being able to like track lineage right and what version maybe of a of a set uh of a of a table that you were using for um training and so really I think this like very powerfully brings to home the whole idea that like Unity catalog really is this catalog for um uh your AI assets

22:16your tables and unstructured data what what's the magic like how did you save the model in in in the UC do you like do you do you use an SDK do you have the race API directly how does it work yeah so uh this is what well we have an SDK and also rest apis for being able to register uh models um with

22:38unity catalog and so um all of this is packaged up uh really nicely and um I can also make sure to link uh we should totally link all of the documentation um so that you all can play around with this um as well at home yeah wait can can you do can you do it directly from mlflow so like using ml flow you just connect to UC is it is it built in oh

23:01yeah well that's that's actually what just happened is that I configured you or I configured the ml flow integration is um to uh use Unity catalog as the model registry yeah so you say you say Okay MF the registries you see and just save the mle here and that's it yeah here let me see if I think I have like a quick uh a quick diagram that I can share of

23:23this yeah so here it is so it's like you have mlflow client right and then it's actually using you're pointing it to Unity catalog is to use that as a model registry uh and then mlflow tracking that's the UI I showed um which just shows you some of the training runs yeah nice yeah and then it's all governed in unity catalog um as well and then here's that three level uh Nam space that I was

23:46just talking about yeah then then you can set the permissions like who can access the the model exactly exactly or you can even you know be able to um you know uh execute this right right or um like in your predictive like um application or something that right as like you know by referring to it as unity. default. IRS or whatever you're

24:08able to actually um uh work with this model in your applications and something important like uh the UI is coming I mean everything is for everything is free of charge including the UI so this is something important because things sometimes it's all out there in the open yeah you can contribute to it make it a

24:30little prettier I think it's actually not bad I think it's not I think actually the like it looks it looks incredible honestly for how quickly I know it came together yes yeah like for like the first UI version I think it's looking good yeah yeah knowing that it was open source on in June like five months I mean not like less than four months later we have a UI we can now and this is what I want to talk about

24:52like you can govern the ml models very smoothly and now we can all have also into this deep integration with spark and Delta and this is pretty much cool because this is what people were asking for like being able to access you can also access external tables manage tables can you walk uh talk a bit more

25:15about this part yeah yeah so um I think I talked a little bit about uh I gu I I SP I spent a ton time like the models and stuff like that um actually available since the 0.1 release was um the core apis are working with tables like external tables um and uh managed tables um both in uh Unity catalog open

25:38source and then also data breaks Unity catalog right because they're both 100% um API compatible and so uh uh the credential vending that I mentioned works on all clouds and that's the mechanism of which you're able to securely access uh this data and then um you can with something else that we just added in the 0.2 release uh was a

26:01spark integration so uh what's kind of exciting is it's like the spark integration for working with Delta tables that are in unity catalog so it's like really like the coordination across like these three different open source projects uh actually so it's like you know like orchestrating all the different releases and stuff like that um but now you can work with you know uh create read write to uh external tables and then read from managed tables where

26:24uh the writing and and creating managed tables uh is coming later because that requires some additional level of coordination with the Delta project around coordinated commits and making Unity catalog commit coordinator so uh that's coming later Delta four huh which is part of Delta 4 the coordinated commits hopefully sooner uh 3 3.3 yeah

26:47but it's also part of four yeah yeah yeah yeah yeah um the uh it's it's super super important because I think that this is what's going to enable us to do things like for example like PCY table commits and and stuff like that so uh yeah tons of exciting things um and uh it's really fun to be able to query uh to connect UC OSS with data breakes need

27:10also to enable on the account console this possibility to quer to be queried by external engines which gives an extra layer of security just to make sure that no ones share like table that that was not supposed to be shared and accessed by the OSS version yes 100% so uh that is something that's

27:33actually out in public preview for for data breaks Unity catalog is this yeah this additional governance uh uh knob for um turning on and enabling external access on uh for example like a schema and stuff like that so turning on external access what this means because by default it's it's not on uh which is a good thing because you know

27:55organizations they want to make sure to like you know um it shouldn't be on by default because we want to really really be cautious about things like any sort of like data exfiltration risk uh and stuff like that so by default it's not on but you can enable at for example a schema level um the ability for external clients which are of course authenticated and authorized and all that goodness um and you have to like

28:17Grant it you know these these principles privileges in uh in unity catalog and stuff like that but um it'll enable those external clients to be able to uh read and write to those uh assets that are in uh in data bre Unity catalog uh or any Unity catalog um yeah so that's a little bit about that extra that extra governance not because you know like I said we have tons of customers running

28:39this in production and it's it's very imperative that we uh roll this out thoughtfully yeah that makes sense and can you U okay I think Quon have a question before yeah no no I was about to say we talk about the late like the latest release which was a few weeks ago do you know what the like the team

29:02and and the open source Community is working on right now and what's going to be the next one that was my question you guys just want to know what's next didn't we give you enough in 0.2 oh my God always want more It's never enough um like you haven't even finished playing around 0.2 yet the uh there's too much um but uh yeah so the

29:26things that we're working on I guess first of all i' also say that like ongoing it's really like high prior for us to continue to work with like all these other uh engines and Community projects and like ml projects and stuff like that to be able to build out um uh Integrations with unity catalog right you don't need to wait for releases uh for for that type of stuff but in the next uh what the team is working on um

29:48right now and what's just like and this is the road map by the way it's all available in like GitHub discussions like we can talk about it there in fact there's like some good discussion about like saying hey like you know can we prioritize some of this stuff earlier and then like you know we just need to put like more folks on that and like you know can how can we get help too right like what what community members like might bring that right expertise to be able to support these different parts of the project um but uh right now

30:11we're working on I think I mentioned uh making Unity catalog a uh commit coordinator um a like server for uh uh Delta lake so that's a lot of collaboration with the Delta Lake uh project around okay Delta Lake it has this concept of like uh coordinated commits where on a table you can say hey

30:33there's actually this uh service it can be like dynb or it can be like a Unity catalog um or Hive um at a store uh that says hey this service is actually going to be the source of Truth for commits um to me not not just the file system commits because sometimes we get this sort of split brain problem where like metadata can be out of sync in the catalog and the file system and so um uh

30:57there's coordinat commits uh so we're going to be bringing that to Unity catalog implementing that as commit coordinator and then what we're going to enable is that's going to enable us to open up um the manage table create and write uh apis which have been um long requested and I know many folks are excited about this I see lots of uh folks in the unity catalog slack asking when is this coming um and uh it's like

31:20okay we're there's a lot of there's a lot of moving pieces here um and so we're working hard on that right now um and and designing some of that interfaces and stuff and then um the other thing that we're working on is uh more of the like gen uh functions um Integrations and capabilities so talked a little bit about this earlier as like we have

31:42functions in 0.1 release that you can govern a Unity catalog uh but uh what we're really excited about is this like really emerging uh but important use case around almost like using a catalog is like a it's like a registry of your AI tools that now we're seeing these like AI agents like people are building these agents that can go call tools so like a simple example would be like you know we know we sort of know or like I

32:06don't know this is also moving under our feet every couple months but it's like you know llms they aren't very good at math um so it's actually easier to have them write a little bit of code that does the math right and then execute that um and so like that's an example of like maybe there's like a a python function for say like I don't know calculating fiscal what fiscal year it is or something like that uh that's a toy example but like that basically these age can actually go call and then

32:29you can also do of course much more powerful things like calling other apis and stuff like that um uh from from these agents and you can register these functions in unity catalog U with a little bit of metadata description about how to use it um and stuff like that and then um what we're uh working on right now also is just like the UC uh like the python the functions SDK and then also a

32:52bunch of really exciting Integrations you can actually see this being built out in the unity catalog repo right now it's like uh like Integrations of like uh Lang chain and anthropic um and like uh open Ai and llama index so all these like notebooks you can see that we're actually building it out there and it's out in the open um all these really exciting things that are going to come to life for being able to uh leverage UC

33:15functions in like this really cool governed um and unified way from all of these like AI uh tools and being able to call AI models and stuff like that so uh that's something that's in the world works as well for the 0.3 release um which the team is heads down on um targeting this this uh this coming quarter yeah and I believe this is our

33:38biggest differentiator to have this AI integration not only the data and this is something important to point out and I guess with all those exciting information you shared with us you already secured your spot for third episode with us which will make you the person who attended or who was invited the most as a guest since the beginning

34:01of this channel I think one year and a half ago oh amazing okay well I'm excited we have to keep releasing stuff yeah I guess by the V1 release you would have been like uh like have like 10 episode maybe that would be good so funny amazing I can't wait I can't wait

34:23to come back and and talk about all the cool stuff any any anything to add Quinton before we close no no thanks a lot like it's like it's really it's I think it's really funny to see how the like the evolution like it was like from hive metastore and then we had to create file format like um Delta in iceberg because Hive had tons of issue and I were like

34:45okay you know let's build something else on top of everything and we will bring Ai and and all this stuff together I think it's a cool story super cool story super cool story um yeah please uh if you want to if if folks are interested uh please like uh go to Unity catalog. or hopefully we can just throw these links up and stuff like that but uh that's where you can find all the links to like following us on

35:08LinkedIn please star the GitHub repo um you can join the slack uh Community um and then I'll also plug that actually next week if you want to learn more about the 0.2 release and what's ahead uh going to do like a live like AMA with uh Denny Lee another uh Rockstar uh over here that I that I work ly with and we're going to do an AMA at the next

35:31Community uh meet up for Unity catalog and that's where you have a chance to win one of these shirts yeah I think if you attend yeah we're gonna give to five folks already convinced maybe 100 folks to join the call to join your at your M amazing great I hope to see everyone there thank you Michelle thanks a lot

35:55and see you next time yeah yes see you next time have a good one y'all bye