Unity Catalog Goes Open Source: Interoperability Standard

0:00it needs to come from like you know having a community built in from the start and so it's something that we were really excited about I mean to be honest that was very epic first edition Unity catalog t-shirt you need um a single unified catalog for governance across your data and AI assets hi everybody so today I'm with

0:22youf and uh together we will be talking about Unity catalog with Michelle thank you so much for you know being here with us um so I don't know can you maybe start by telling us more about yourself what you do a air breaks yeah absolutely well first of all very excited to be here joining you all um a little bit about me

0:46I am a product manager at Databricks so I work on a couple different things I work on open source Unity catalog which is what we're going to be talking about today I also work on Delta Lake and so very familiar with both um open source projects and and and growing and building up those communities and I'm a product manager and I also am based in San Francisco and happy to answer any

1:10more questions there about all right awesome so one of the big announcements at the last dat n subit was the release of unity catalog as an as an open source project uh which I believe mate did live on stage so releasing the catalog just now C can you tell us more about Unity catalog like what is it at a high level uh which challenge does it solve

1:33and and you know why people should should care yeah okay so I guess I'll start really high level with unity catalog and I'll talk a little bit about just like maybe what what cataloges are solving in general so catalogs what they're solving is first we have this concept of like open data Lakes right so all these companies they have tons and tons of data maybe they have a bunch of different tables maybe these tables are

1:56stored in a bunch of different multiple different open storage formats their um uh containers or buckets maybe they'll have images and unstructured data maybe they'll also have models which are leveraging um all of that data uh in order to provide insights um or power like you know real time AI applications and stuff like that and so organizations

2:20they have all these data and AI assets and then what a catalog does is it helps organizations um with metadata management which is a very very very dry term for basically how do you um Power and govern all of this data and AI assets that are in your lake house that are within your organization um how do

2:42you actually um get insights from it um Power Discovery things like observability and in lineage and auditing um how you make sure that the right teams of people have access to the right data um and that maybe we're classifying data that's sensitive and Stu stuff like that and so catalog is really at the Nexxus of all of these things um helping

3:05organizations govern and manage their data links all right so so so it's about uh getting access to the data and all the AI um making sure we can do discovery on top of that uh all the role permission and all these kind of layers um so can you tell us what's Unity because Unity catalog is the catalog which is going to stand on top of kind

3:28of computer engin like you could have Doug DB dat bricks anything else um so maybe what's the catalog doing compared to what's the engine doing yeah so let's see I I realized I said probably like a bunch of different a bunch of different jargon um and so then maybe I can like start with uh taking out some of the

3:50jargon and saying okay what are you trying what what's an organization trying to trying to do ultimately at the end of the day I'm a user saying I'm Michelle I'm trying to like query some table I need to be able to um first I have I have the table name maybe my data scientist told me like hey here's here's the right tables that you need to query you know there's some data on users and

4:13there's some data on like I'm say um I used to work Airbnb so say I'm trying to like query some reviews data or something like that on listings and um C the role of the catalog it's not the query engine itself it's the thing that the query engine talks to where I can say I can give the uh query engine the table name I want to query the user's table I want to query the review's table I want to join these um I want to select

4:36these columns and stuff like that and also I'm um trying to query this with the permissions of like Michelle who's like a business user at Airbnb and um the engine I might use would be say something like spark or trino um or Presto or something like that and then that engine would talk to the catalog which is managing all of um

5:01in this case say like airbnb's uh data and then sharing that table name actually get back metadata about that table and then be able to go and actually access like the underlying storage to get all the right files and query that in the metadata understanding the schema and stuff like that and then be able to return back my actual results

5:23um so I did so the distinction here is that the engine um on it own wouldn't necessarily like have all this context about oh the governance or what user has which permissions on which tables and um what's this latest uh schema and stuff like that and and so that's why you would be talking to the catalog in

5:45this case first to make sure that you have all those right permissions and that you're um accessing the right table and stuff like that and accessing the right set of files that power that table underneath so does that does that kind of um hope I could even like go a step further um in terms of like you know okay how would you how would you describe like Unity catalog for a non-technical person

6:07too Michelle uh what inspired the decision to open source uh Unity catalog yeah okay so great question um something that we've really been um pioneers of and have seen in the industry is this concept of open data Lakes which I think I touched on earlier but it's this idea that you know all of your data it's in a

6:30open storage format um stored in a bucket or container that you own and control and so you will always have um full ownership and control over your data it's stored in an open storage format so there's no sort of like lock in into these proprietary formats um and so that's the concept of data Lake and then what data BRS is pioneered is this concept of like a lake house so it's

6:52like now that you have all of these uh files stored in Blob storage and these are actually powering all these like structured data applications and CR critical business um applications and stuff like that um what you need is this concept of the catalog and um that's what powers The Lakehouse and it manages

7:14all that metadata and stuff like that what we've seen is that in the real world users and organizations and Enterprises they aren't using a catalog by itself they're not just using one solution in the real world they might be having um on premises data right they might be having um tables or data that are managed in other cataloges um third

7:37party systems other external systems and um so the real world architecture is quite complex and what we've seen is that this this idea of like metadata silos um kind of forming without this sort of open data Lakehouse and interoperability between all of these different um cataloges and solutions and so the decision to open Source Unity

8:00catalog is stems from this idea of okay well we want to empower this Vision this is something that we hear a lot from our customers so product manager means I talk to like a lot of customers every week and something that every customer has reiterated is this idea of wanting an open data Lakehouse where any engine um any compute engine any tool that's the right tool for the job is able to uh

8:22access their data that's in their data Lake um without needing to like mess around with like gluing together all these different Solutions and so there's two things that we wanted to do with open sourcing Unity catalog which is the first is creating like a standard open standard for interoperability for these engines um and for other cataloges to be able to talk to Unity catalog to be able

8:47to um access all of your data and AI assets that are in a single place and then I think that the second thing that we're trying to do with open source Unity catalog is that you know this is something that we have like a strong thesis around so we started building Unity catalog several years ago around this thesis that you need um a single unified catalog um for governance across

9:11your data and AI assets so not just tables like structured data but also unstructured data like images or videos or you know raw PDFs and documents and stuff like that and then also maybe the things like um you know your AIML models or your your functions like you know

9:33those geni agents that you're building with all these different data sets and stuff like that also need to be governed in the same solution and we don't see that there's you know anything else that's open like this in the market today and seemed like a big gap and to us it was a clear decision to take the thing that we've been building internally for the past several years

9:56and actually um bring that to the open source community so that we can evolve it um out in the open and also create that standard for interoperability and and and break down some of those metadata silos that makes sense and just to go back mate um made the repo let's say available uh to everyone on the stage

10:18and that was an epic uh I think one of the best presentation I've ever witnessed and that was real by the way it was very nerve-wracking yeah yeah I mean that's for sure like if something went wrong like yeah but I mean to be honest that was very epic and I've seen like people talking about it after the presentation for very long time I've seen record like people like uploading

10:41some like downloading records and doing some Cuts just to publish this this five seconds where mate is announcing that we're doing it now on the stage and just opening GitHub go to the d section and make the repo available to everyone and and the question was like I've seen that you announced several partners and was it uh like how hard was

11:05it to keep it sex secret because I know that you've been working with several Partners you've been like developing but and it was meant to be announced at the data Summit how hard was it to keep this secret secret um that's a it's a funny question that you asked it's very true so we were working with partners for you know months months leading up to months

11:27leading up to the summit because you know an open- Source uh project it doesn't just start with you know a sing a single group of folks saying all right well you know now let's just make the repo public and then know okay I guess we're I guess we're done and it's it's just this group of folks it needs to come from like you know having a community built in from the start and so it's something that we were really excited about is working with a number of Partners um both uh like AI companies

11:53which I think is really interesting and then also you know compute engines and stuff like that both open source and even and um proprietary and stuff like that um where the key is we had um actually been meeting with folks before of course we were trying to keep it secret to not steal our own Thunder that we're were going to open source Unity catalog it was wasn't hard until I'd say

12:16like the week of data and AI Summit where you're balancing all these different things so actually at the at data and AI Summit I think like Yousef and clenton you were there we have this like open source mini Summit this other secondary track that was happening at the same time as data and AI Summit um that's really around like spark open source Delta Lake um mlflow um but we also had a track for uh

12:43open source Unity catalog which we couldn't say right so why would Unity Catal it's very suspicious Unity catalog is part of the open- source mini Summit track and so we had the session so we invited a bunch of Partners we had this community kickoff it was amazing um but then also we were trying to keep it secret we called it I think like a just private session or private track or something like that so that people wouldn't see that it was Unity catalog

13:06although people kept trying to poke their heads in um but it was hard to that was that was that was that was probably that was probably the hardest bit is like you know you're actually there live with all these people um in in our community and then uh we're trying to say oh you know oh oh Michelle what are you what are you doing here and it's like oh no we're just uh we're just here in the private session

13:28don't don't mind us having to keep all the doors closed and stuff like that but yeah it was fun though and the kickoff was great all right amazing so you you mentioned a bunch of Partners working on UC can you can you give us a few example of the main partners and maybe the M the main system already starting to speak to speak UC see yeah um well let's see the some of

13:54the partners and especially the ones that we went to for example so actually we um I should mention we donated Unity catalog uh to the Linux Foundation Ai and data foundation and so we went to um LF and data with a number of Partners um already in support of unity catalog

14:17and so examples of these Partners um are friends at at Daft like the multimodal query engine um poppy graph this like graph query engine over open data like formats um seller data the folks behind star rocks um Lance um Lance DB which is the vector database yeah and they also work on the

14:41Lance um storage format um which is more for like AI applications um and then uh we also worked with a number of Partners like um like uh llama index and laying chain and stuff like that on on on working on Integrations and stuff like that from the start and so tons of Integrations have already uh come out in fact um I

15:06want to say like the first maybe the first two come out there's I would say there's almost two that were competing is uh um Daft they built a client integration and engine and then generated python client within like a two hours of the repo being live um and then also that same day um Kyle our friend at at one house um the folks with

15:30the POI X table also built out an integration and demo all in the same day and not only that he also even demoed it during his talk at dat Summit later that day after mate open sourced it live and so I mean as you can see first of all it's really easy to build with with unity catalog and then um second of all um there's tons of tons of momentum uh

15:54in terms of folks being eager to to build out Integrations I can speak you see as you put it and um since the release of the first version uh what were the improvements that were added yeah okay let's see so the things that I'm most excited about I

16:16think I mentioned all these Integrations right so the there's one thing I mentioned there's like two parts right so the open catalog standard this for interoperability and then also like this open source uh server implementation of those apis and the CLI and stuff like that and so I'm really excited about the Integrations that are coming out because these are things that are able to speak you see um and

16:39uh uh make it super useful for users and customers of data Brooks and anyone who's standing up their own Unity catalog um or any catalog that can that can speak this same open protocol and then the second thing that I'm really excited about this just went live last week can check it out at the um at uh

17:02the GitHub um for Unity catalog we have a new UI so I mentioned that we have uh we had started off just with kind of like the server and the CLI and um the open API spec um and then now we have a very fancy new UI for being able to actually explore um all of the

17:25different assets that are in your catalog um and there's lot more to come soon on on that front it's pretty slick you should uh give it a spin if you check out the GitHub repo yeah definitely We'll add and I I think that's also lead us to the next question what if I want to contribute to the project or what are what should I

17:49do yeah okay so there's a number of different ways that you can get started um I think it's like very uh I think I think what folks maybe don't see as much is that there's so many different ways you can get involved in the community it's not just that you need to contribute code for example so a few great ways to get started is first you

18:11should go check out Unity catalog. or I see where also probably put all these links in later but check out un catalog. it has links to all the things I'm about to mention which is you should join the slack community so getting involved so they're even just asking questions they' be helping to answer questions um the uh being able to just like um ask for

18:33like oh what are the maybe the upcoming client Integrations or stuff like that learning about the road map the second way you can get involved is you should absolutely um uh join the GitHub uh or uh repo actually oh star it by the way um and uh you can actually go check out like issues and discussions um there's a number of

18:56issues that are tagged with like good first issue and so maybe you're someone who's new to the community or to contributing you can start with one of those that are Ted with good first issue if you want to start just contributing code um and then another really great way to get started quite frankly is actually helping to us to continue to improve and flush out our documentation and so if you go to the GitHub today

19:20you'll see that there's a road map with a number of issues that are tagged as Al good first issues that are tagged with um hey here's where we need help with stocks um also you can get involved in the UI as well there's a number of um issues tagged there and so you can actually go in and start commenting and say hey I'll grab this um and would love to would love to take this on and so there's a lot of different ways that you

19:42can contribute um even if it's not writing code yeah and I think we have um open our that everybody can attend right yes oh great question so uh yes there's a few there's a few different um additional ways you can get involved even for serers we have bi-weekly

20:05Community meetings um these are both live streamed and you can also just join through um the uh they're they're they're hosted with the Linux Foundation like Ai and data foundation so you can join through there and we have these bi-week Community meetings we'll talk about roadmap um had a number of Partners common demo um Integrations or even just how they're thinking about

20:27building with unity cap log and it's a great way to um ask questions get your questions answered um and then also see what's top of mind and upcoming for the for the unity catalog community and and talking about the road map how does it work like if someone um has an ID or you know want to change something within Unity catalog do do you run votes like who decide what what gets

20:51published and created yeah so it's a great question so the best way to um say you have something uh that you want to add to the road map or an idea or suggestion how you would get started is you would um create a GitHub issue maybe start a thread in GitHub discussions and um or maybe find an issue where someone

21:15else already kind of like maybe someone else already had a similar thought and then you can actually go engage there you know um uh uh like the issue engage on it debate um ultimately like we are a Community Driven project and so the way that we think about um our road map and

21:38Milestones it's about okay if we're looking at okay what are those like critical next things that we need to get done also where's the discussion where's the um where's the discussion the community headed at and there's a lot of different directions that we could take and so right now I'd say that we're still in the kind of like the crawl and walk stage of Craw crawl walk can run so you know for example we have people

22:00asking already about like oh Advanced like you know AI classified um you know pii detection and and tagging and it's like okay we're GNA get to that maybe uh maybe later and you know first we need to start with the um hey let's go build out this amazing UI let's go make it easy to deploy Unity catalog you know add a plugable um back

22:24end all of those things and then you know adding additional things to the open API spec and implementing those and so we're working through all those backlog of um of issues um but the best way to get involved is to just start engaging in discussions and and start engaging in in issues and in the slack community and maybe maybe you will get this cool t-shirt for the yes oh we

22:48should actually plug the uh yeah this this t-shirt this is a first edition Unity catalog t-shirt from data and AI Summit yes so if you attended uh one of rsh and I rsh is the tech lead by the way on kog if you attended one of our breakout sessions you might have gotten one of these um and if you didn't we're going to be doing another run of a

23:11ton of merch not just shirts we'll do mugs we'll do socks we'll do the we'll do the whole thing I can't wait for the socks actually and um it's going to be for folks that are super involved in the community and contributing and um hope to see you there so I have a question for you me and queen are active member of the community right yes of course of course you are I

23:35see you all over the place just for those who missed um so actually Michelle and um her colleagues did run two sessions at the data Ani Summit uh so those are deep dive sessions we're going to put the links uh in the description of the recordings as those now recorders are now available on YouTube to everyone feel free to watch

23:57them those are deep dive once they explain how it was architectured and we also I think have a demo there think with I think duck DB and UC if I'm not wrong if I'm not wrong that was the demo of duck DB and UC so we'll have description on the recording the description of the recording and any any

24:19other things to add Michelle for closing um nothing nothing else nothing else to add I guess I'll maybe I'll just plus one that plug on on a breakout session we did a number of demos and and I I do have to say uh again like my gratitude for the partners in the community on like how how engaged um everyone has has been um and the

24:43excitement that we've seen so I'll plus one the plug on the breakout session which is that we did demos uh not just with duck DB but also with um uh trino with uh unstructured IO uh working with unstructured data and volumes and then also with Lane chain working with functions and so download all the different capabilities of Open Source Unity catalog um and so definitely check

25:06out that session and then um let's see I'll I'll also say that uh the reason I'm so grateful and the reception too is that um one fun stat is that we've reached I think one 10,000 stars on on GitHub in a matter of days of the project launching uh which is setting it's a it's setting a record at least

25:28here at Databricks for um the amount of love and and reception for uh a new open source project that we've launched um a smashing records I should say um and so really excited to continue to grow on that momentum let's say let's seek for the 5K and then maybe why not 10K and more and more so let's keep us number

25:52one in the open source love it yeah well yeah it taking over the world in the number of uh of stars yeah so connect on slack and go on KB and you know add a star on on on the repo let's let's pin the numbers thank you thank you Michelle for um being our guest I I was about to say

26:15host but being our guest uh was uh super interesting to have all those uh crunchy details on UC open source and let's see you maybe in six month we'll have bunch of new new features maybe to show to the community oh we definitely will next time I'll come with uh with even more

26:37demos thank you Michelle thank you for having me great job thank you bye bye