Open Source Governance: Balancing Business and Community

0:00the data and AI itself is your company's asset you have have candra spark like was so many of them and then you had big companies behind them uh creating their own product hey folks today we have Den Lee as a guest he's a staff oh he's a

0:22principal developer Advocate at data breaks he has an extensive background in data engineering analytics and distribution systems he has over 19 years of experience in the tech industry he works previously for uh Microsoft and now he's working of course for data breaks he's coor of learning spark second edition and also the upcoming

0:45delal Lake the definitive guide and of course a longtime contributor to Apache spark ANL flow I think your resume is so long I think I'm gonna stop here hey Den hey Quinton how you guys doing hey Den thanks for being here oh I'm glad glad you guys invited me all right so I think today we wanted

1:07to um start talking about kind of the open source ecosystem and and how well maybe as a first question as an open question you know what's what's open source these days because I feel like I can create any kind of GitHub repo uh and call it open source but then you can also go with the biggest project like from you know for example Apache spark and going with the Apache um

1:30ecosystem can you yeah can you start by talking about that and and and and see what's I guess what's open source and when do you need to go with systems like Linux fish and Apache oh sure sure so thanks for thanks for asking I mean the question is doesn't necessarily have the most straightforward answers because it

1:52can be defined by different people in different ways but as a general rule open source is about somebody initially St with the idea that they want to share their code with the community and they believe the idea is that together as a community we can make the codebase better uh we can take account of a very diverse set of use cases backgrounds

2:15ideas merge them together and improve the code base right that that in a nutshell is what open source is about um from a personal perspective or from an individual perspective there is also the ability to Showcase your own technical ability right and to Showcase what you happen to know which actually is quite important when it comes to the job market these days because it allows you

2:37to basically showcase your capabilities your technical abilities whether it is by which you talk about the codebase that you talk about like you know python rust SQL or even your ability to communicate with people within slack GitHub discussions uh email threats now how that relates to when you start off you're starting off typically small

2:59right and then basically over time you're going to say that I need to actually have some form of governance structured around these open source projects and what we mean by governance is this idea that okay we don't want a single company to control or individual for that matter to control what that technology does because it is a shared resource by the overall

3:22community so there's organizations whether it's the Linux Foundation the Apache software Foundation Focus uh lots of different organizations like these those are the ones I I typically work with so that's why typically I bring up those three but the reality is there there lots of very Solid open- Source foundations okay and and the whole premise here basically is to Simply say

3:45that okay I want a governance that says these based on the contributions by said community members they then are um pm C's or tsc's or maintainers whatever naming structure you want to use that basically ensure the quality of the codebase ensure that

4:09we're following you know Apache license or MIT license or whatever licensing that you're using to share Set codebase uh ensuring that when a individual is contributing code that they follow um standard legal and governance procedures that state very clearly that this codebase is actually given to the community not just like company X

4:33decides to donate it and then later on claim oh you're not allowed using it right the the whole premise is to provide governance uh and a legal framework by which to uh share this code base and and so that's why you typically have these type of Foundations it's to basically formalize that structure announce what that structure is so

4:55everybody that's involved with the project understands what is it they're doing uh how they're contributing and how they themselves also will be recognized for their contributions because it's important aspect that when you're contributing to an open source Community it's not just about you helping other people of course that's important but it's also from just a again again from

5:20an individual perspective how does this help your own personal career how does this help you rise the ranks whether you want to be a really solid distinguished engineer to you want to be basically a CEO of a company right the whole gambits there and a lot of it comes down to your ability to not just

5:43write code but to communicate about the code and the design and the architectures that you work with and again all of that is then that framework of how you share and what type of information is being passed back and forth and so um so a couple of extra question on that the for what do you see being the like what's the difference basically on on multiple governance

6:05system like the Apache governance the Linux Foundation governance I know everybody kind of know Apache I feel like for some reason I don't know why the Linux Foundation is less kind of well known established in the collective mind I would say what like what's the difference between them so Apache software Foundation lens foundation and for that matter num Focus

6:27they all have slightly different board struct structures they have slightly different goals right in mind so for example the part of the reason you probably hear about Apache software Foundation certainly because we're all three of us are from Databricks then you we've heard about Apache spark and so because of aache spark that's literally the number one thing we think about right but don't forget the Linux Foundation the whole reason it was

6:49created is because of the most widely used operating system in the world Linux right so it's not like the Linux Foundation is not a known Factor quite the opposite is it's actually larger than the Pache software Foundation I think cuber is also on the Linux Foundation no am I am I wrong I'm sorry which one sorry kubernetes that's right kubernetes is also part of it pytorch uh

7:12that's why uni catalog and Delta Lake are part of it there's all these different other systems right because as linkx Foundation has expanded it definitely pushed into the AI and data space in fact uh both uni catalog and DTA L are part of the Linux Foundation Ai and data project specifically because of that oh I'm sorry and also ml flow right so every one of these foundations operates slightly differently and

7:35candidly each one of them has their own set of unique politics and I I don't mean politics in a bad way everybody always assumes politics especially in this time day and age they assume politics is a bad thing it's like no no no no this isn't an implication of bad or good this is very much a statement of the context of how what are the people's end goals right and so for example some

7:59people people will state that lenux Foundation is a little bit more in tune to the idea of mixing business like running a business and also open source and again some people agree with that statement some people disagree and and this it's a healthy debate so I'm not going to actually try to say one thing or right or wrong right I know that for example early on part of the reason we went ahead with went uh with Delta Lake

8:22for and ml flow originally uh and more recently of course uni catalog all three of us are wearing those t-shirts today um is just because of the context that um Linux Foundation offered legal services offered marketing Services project management Services right from the get-go and so when we were building up a a governance framework we could actually literally utilize their lawyers when there were uh trademark potential

8:45trademark violations uh again nothing egregious but there were actually in some cases and so no problem we actually had legal support to do these type of things and and do you know how like who who is paying them is it is it sponsorship do you know it's sping correct right basically as a general rule Li foundation and Apache software Foundation is a bunch of sponsorship mechanisms right they're various

9:07different companies that basically get involved to basically cover the cost of running these different systems and so basically again I'm sorry I think they also have trainings like uh that's correct yeah they have a very exactly they have very extensive training system for example pytorch has recently been pushing out a lot of training so that's probably some of the emails that you've been receiving as pytorch is part of

9:29Link Foundation uh we're about to do the same thing by the way so you haven't seen nearly as many as us so maybe next year you'll start getting a little ticked off from those emails as well that's amazing and and can you so can you give us like a in like in less than one minute the typical kind of setup and governance system for these project like we have PMC we have committers how does it typically work no problem so from a high level and you know if anybody who's

9:52a part of the Pache softw Foundation May correct me because I might not exactly get it right but as a general there's the idea of you have a uh contributor person who contributes code you have a committer person who actually can commit code who's actually allowed making merges to the GitHub repo or to uh Maven or whatever else right and then you have the PMC these are the um the project

10:14management committee like they're the ones who actually can sort of control uh this make the decision factors of how things work of the particular project so that's typical for an ASF project not always but that's close enough for a general rule okay and Apache spark more or less operates the same way um for Li Foundation projects uh there's a little bit more variety in the structure but again general rule in terms of some of

10:38the projects at least that I'm personally involved with um same concept of a contributor you contribute codebase you have a committer you can commit code uh but you technically there's a maintainer okay that actually is the one that maintains the whole codebase the whole cold quality that's almost analogous to the idea of a PMC member for a particular project and then in the

11:01case of uh the way un uni catalog Delta Lake and mlflow work uh we have a TSC as well the technical steering committee they're actually not involved day-to-day they're basically sort of like almost like the board of directors of a of a particular project and so that's a higher level that basically every so often we're going to go ahead and talk to them are we following the governance structures yes we are no we're not things of that nature but the

11:23maintainers are the one that maintains the projects now again each project has a slightly different variation of what I'm just saying so this is more or less the way um the ones that I'm involved with pretty heavily sure but as a general rule that's more or less the context okay that that makes sense and then each each major OSS

11:46product has some big companies behind it so how do you think they find this sweet spot in order to ensure governance is well balanced well see here's that's that's that's one meant by the politics of any open source project right the reality is every company that gets involved

12:08wants to be able to guide the project in a way that is both at least in most open most successful open source projects I should put this way they have this balance of trying to figure out how to do what's both right for their own company and also what's right for the community in general right and it's a

12:30really really difficult thing to do right if you look at this day and age where a lot of companies are restructuring their licensing where they'll actually make basically the legal Le States things of the nature like oh if you're larger than x then you you have to purchase the software or you have to

12:52provide dividends or whatever else in a nutshell the whole context is some of these companies feel or these cut excuse me feel that they need to do it to protect themselves from any of the hyperscalers and this is not again I'm not judging this is not a right or wrong this is this is what they felt they need to do and they need to do that because again it's that balance between the

13:14company and or companies and the open source Community itself and the reality is it's really really difficult to find that balance right that that's what it boils down to there actually very few companies that are truly successful in building an open-source technology sharing that with the wider community

13:37and ensuring that they are also successful in both the company itself and also the community this is why you actually see a lot of us like the one crucial thing that I'd love to just do a shout out to data breaks is that sometimes yeah people just associate myself you know my face with the idea of osource but the reality is that this isn't just I'm certainly not the only person quite the opposite there's a lot

13:59going on right from the CEO right from the East the founders of the company all the way down to the key Engineers that are in the company and even our sales folks even our field folks they all understand this idea that we actually have to find that right balance between open source and Company priorities and the reality is that it has to be built right into the culture right from the be right from the beginning it was a little

14:21easier for us just because that's where we were we started from that was our route right so it's a little easier for us to make that statement but it's not easy okay so it's easy to stay it of course but it's extremely difficult to find the right balance and and so on that topic how do you see the like the kind of OSS um OSS ecosystem in the data currently I feel

14:44like if you if you go back in the 2010s you know 20 between 2010 and 2020 I feel like we had tons of new um you know OSS project in the in the in the data space like you would have Kafka you have M candria sparking like it was so many of them and then you had big companies behind them uh creating their own product and I feel like now starting

15:08in 2020 we see less of that I'm not saying we don't have any open source project like and and there Lots UC and and and you have Doug DB so we still have some but I feel like something changed do you have the same kind of perception oh absolutely but there's actually a multitude of factors it it isn't one thing right so for starters the reality is some of the top

15:32developers who created these open source projects the ones you mentioned and many many more right what happened is that all of them recognized that they wanted to make money too and I don't mean like you know they're trying to be like you know uh multi-billionaires or anything like I mean just like they need to make a living okay and so they all be either created companies or became part of those companies all right any any open

15:55source project that was remotely successful end up making that shift just like the on you mentioned the problem is multifold first of all not to get all economics on you but the interest rates are have increased okay so in other words borrowing money has become more complicated more difficult if boring money is more difficult VCS are not inclined to give you as much money so unless you're saying gen gen gen they're

16:17not going to give you any money okay so so there's that minor problem okay number two most of these companies that have built the infrastructures we always go through these Cycles when it comes to Technologies right I uh prior to this I was at Microsoft and I still remember when I was at Microsoft I was part of I help create a SQL Server database at that time people I'm probably too old for y'all but like we were dealing with

16:39this idea of multiple relational database we actually were should we build object-oriented databases and there the fragmentation was amazing and what happened over time basically we started consolidating these different Technologies different systems realizing okay maybe objectoriented databases wasn't the best idea again not trying to debate anybody that's just what was decided relational database is what um

17:02is what what was dour and then after we all decide RBS was the right way of doing it then we started consolidating all these rdbs systems into just a few right we're basically seeing the same thing happening again when it comes to the data aspect of um the uh open source Technologies right so in other words that's that's the normal part of the waves and the other key thing right so

17:26the third key thing of course gen right when when chat gbd came out finally everybody's understanding F people finally understood just because they had a like little nice cool messaging app that says I can chat with you my goodness now I'm going to make a billion dollars because I can go ahead and create a cool chat gbd option obviously people are recognizing maybe it's just a tad bit more difficult to than to do

17:49that but that's what's going on like a lot of the a lot of the really cool developers they didn't actually have to create open source Technologies to go ahead and showcase what they did they literally created said I'm a PhD in something to do with Gen with AI bam VCS were like here's a bunch of money and boom now they have companies

18:12so you see a lot of small very strong very smart people they actually were able to bypass that stage and and jump into directly creating companies so right now sure we all hear about coher anthropic and open Ai and Databricks and all these big ones sure but there's hundreds of small little ones that are full of really really smart people and guess what just like I mentioned before

18:35about the consolidation yes over the next few years because the interest rates still stay relatively High people are recognizing oh I have to make money come on Microsoft in their own Securities filings recently had said oh yeah all that investment on hardware for Gen it's going to take us 30 years 30 years to get the money back right you're going like wow so everybody's recognizing okay maybe

18:58maybe we need to stop and recognize how we actually can make actual money on these things so there'll be consolidation as well this is to say that it's all waves so do I expect to see in the future some other big push for Gen or other picture books for like Quantum Computing or blah as a general statement yes right because

19:21what it comes down to is that open source is still an extremely good way to Showcase your ability is still an extremely good way to build up a a a diverse set of people diverse set of ideas to work on an individual project to go ahead and actually improve it right it's still the right way of doing it the question is when when will things

19:44calm down enough because right now it's all about gen you know so right now this this recording will go well for the next year or so and then in about two years from now it's like Jad that's so old right so like yeah so let me at least plug that in real R from the get go so two years from now when people are listening to us they like oh yeah yeah we understand that this is this is a stage this is a phase for all of us right now so and then at the data and

20:09Summit we did unveil uh UC open source which is not not only a data but a data and AI catalog so this is something important and I've noticed that multiple companies were part of the the Doss project so how did you handle or manage the communication between those multiple companies because I've seen like an

20:32extensive list oh yes I mean in any any project like this right and UC thank you for calling out that UC you know came out of the gate with lots of different companies involved in reality is it's a lot of conversations that's it I mean sure you can do emails sure you can go ahead and uh uh put up gith up discussions of slacks but the reality is

20:54a lot of this is based on personal communication personal conversations right part of the reason you know my the or that I'm part of is called developer relations is literally as defined it we relate to developers right that's literally our job and the number one thing about relating to developers is that it's a developer talking to another developer right in other words it's a

21:16coder talking to another coder right and so the Crux of it is that this took time because many people I'm not the only one obviously right were talk having lots and lots of conversation to the cloud hyperscalers to all these different AI companies all these different data companies because we all agreed that we needed a new standard

21:40when it came to how to manage and how to have governance around all of these different Ai and data assets both right that's where it was became difficult we're Rec we're all recognizing governance actually ends up becoming so important for the success of what you're trying to do that if you don't have it then basically it does literally do I know

22:04who has access or have I prevented uh authorization or access rights to a particular table or are they allowed accessing that particular model if they're allowed accessing that model model are they allowed accessing the hyperparameters or do I have lineage that tells me for this large language model when I ran you know 1,000 shot or

22:26CPT what was the original data source what was the synthe synthetic data that was generated from that data source what were the hyperparameters that were used into that what governance uh was structured around that so we know exactly what data sets and what parameters and what uh um governance was allowed for all these things and ultimately that structure is become

22:48is becomes important because people are recognizing the one the most important thing about data and AI which is the data and AI itself is your company's asset right it's your company's IP and so now that more and more companies or organizations are recognizing that the data and AI assets are your IP they sure

23:10as heck better go ahead and place governance around it to ensure the quality of it and what about like the current stats like on I've seen that we have uh don't remember the numbers or like how many stars we have on kiub right now and how many people have been uh by participating and committing in the

23:32project oh yeah so I mean today is October 21st and I'm probably using numbers that are probably closer to October 1st but we have like last time I checked something like 2,300 Stars we had more than 70 contributors uh to the project um yeah I mean it's it's growing really nicely I think um ail she went ahead and had recently written a Blog on uni catalog IO at

23:55least as of today's recording recently um noting the fact that the rate by which contributions and uh stars to the project to UNI catalog project was actually technically higher than what even Apache spark did which is pretty sweet right um but by the same token different times different days right so I'm not that's not a quite Apples to Apples comparison but by the same token

24:18still a nice nice little shout out yeah yeah so talking about open source uh their bricks aced U tabul which is the company behind B and their bricks is also behind Delta uh it seems like there is a willingness to kind of stop the war between all these open source formats um well you know all the open source ecosystem can you can you share your

24:41point of view on what's going on here and and you know maybe how do you see the next few years oh yeah yeah I mean look the reality is that all for that matter all three like let me give you some context all three for lake house formats right Delta Lake Apache Iceberg and Apache hoodi we had already started started basically going this is a waste of time in different forms like we all

25:03had conversations off the side going this is stupid let's not do this and because if you really think about it underneath the covers we're all working with most primarily working with parade data sets anyways okay par data files so the only difference between what we're doing is basically how we manage the metadata okay so for example Delta we created uniform and part of um Delta 3

25:26and Delta 32 we in included uh Iceberg and Hoodie respectively with part of uniform where basically when you create a Delta table it went ahead generated the iceberg and Hoodie metadata as well at the exact same time same parket files just different metadata and then boom you're good to go right um the uh our friends are over at one house they went ahead and created

25:48Apache xtable table Yeah whole purpose is basically an open source version of that too so cool we'll we'll figure out which how to uh how to convene those right this is good people having good ideas we just happen to all have at the same time okay cool it is what it is and which in the instance like you talk about tabular so let me let me roll backwards a little bit and talk about it from a data BRS perspective before I

26:10talk about from an open source perspective all right from a data BRS perspective our customers re they all chose a particular lost format whether it was uh Iceberg or Delta or hoodie doesn't matter they all chose it they all had good reasons and in the end we all all everybody agrees no matter how much there was a quote unquote war between all of us we all

26:33agreed you better go on a lak house format first right because you needed transactional reliability on your data sets so we all agreed on that right from the get-go okay so what we recognized afterwards like while they're literally exabytes of data a day processed at Databricks using Delta L there are plenty of customers that rightly understandably chose to use

26:56Iceberg so how do how do we reduce friction how do we make sure it's easy that when they made a very good decision to use Iceberg to make it easy to work within the context of Databricks there you go well that's a simple answer on why basic data brecks went ahead and acquire TBL to basically work closer

27:18even closer with the iceberg community in the end we've all known each other for quite some time right we all worked with each other in the Apache spark commune to begin with so this is we're all a known factor with each other okay so so that that's from that the company perspective now let me roll back and talk about it from the open source perspective again we've all been working together one form another from Apache spark we've all been working together

27:40from the standpoint of Delta Lake iceberg in terms of either Waring with each other quote unquote or working closely together uh we borrow each other's ideas for that for that matter why because we're the same set of people it's the same crew of people that I've all been working together in one form another for quite some time so in the real it is we all were starting to go toward each other okay so I I'll use a

28:04simple example um when Delta started we basically recognized the fact that we couldn't use the HMS right it it basically there just we were breaking the system down the HMS literally was falling apart when we were trying to uh process pedabytes of data then but in the case of Iceberg Iceberg came from another way which is say it's a lot simpler to use an HMS and they're

28:26right no no like that they're absolutely right if you had an HMS right from the beginning it's a lot easier but at the time that we went ahead and did this we could not process like uh the companies that were processing data and were literally talking about pedabytes the HMS would just fall apart now both of us both Iceberg and Delta came separately to our own conclusions okay we could

28:48rely a little bit more on the file system which is exactly what Delta Lake does in order to ensure transaction reliability and also we need a catalog much better than the current HMS systems in order to be able to handle multi-table transactions which is exactly what Iceberg defaults to so just that little example all sudden Delta and Iceberg are getting closer and closer together in which okay we're actually

29:11trying to take advantage of both of these scenarios we came across that again just by actually processing the data the more and more Iceberg process the more and more we process the easier we were trying to Del we were trying to get Delta to become the more production worthy production a that Iceberg was getting and it already is yeah the reality is we started recognizing we're doing the same thing like that's

29:36basically what it B bows down to like when you we because what what ends up happening is that after all these years the combined knowledge of the idea of like how to make it easy for everybody to use and how to actually handle pedabytes to exabytes of data we all all realizing oh yeah we actually all need to do the same same thing yeah so I I can't tell

30:01you pretend to you that like you know I know exactly what's going to happen but what I can also say is that there's a lot of Delta people there's a lot of Iceberg people and for that matter there's even a lot of Hoodie people we're all talking we're all working together and as we progress we're getting closer and closer so I I suspect that we're going to get closer together just because of the nature of yeah we're actually all trying to solve the same thing which is provide transactional

30:24reliability to your data make it simple make it easy so that way we can actually all solve the fun thing let's go ahead and do chat gbt on everything you know that type of thing yeah we have a nice Unity catalog on top of that of course of course we need governance on all of it exactly all right awesome well thank you so much Jenny uh I feel like we should do it more fun and then you know get you on another one so we can deep dive maybe

30:47more into UC or maybe more another you know open source project oh I would love to I Really honor you guys thought of me and I appreciate the time and uh yes absolutely whenever you like to chat more boom let's do it I I think we should wait until the book is released so we can promote the book like hey we have a new book Delta Lake definitive guide get started I mean yeah yeah yeah the book actually will be released in

31:10end of November actually public like yeah okay nice yeah so so you you yeah it's it's only like four weeks away basically I'm waiting for it I order it one month ago so I'm waiting for it thank you very much at least at least we sold one book yes One customer at least I One customer

31:32yes thank you thank you so much Denny for uh for joining the talk and see you very very soon I guess yes meru appreciate it guys thank you byebye