Governance and AI: The Future of Enterprise Data Platforms

0:00things that were changing in the world the move to cloud computing all right so search sh policy with ACL and making sure the data can easily be leverage governance Is Not A New Concept multiple solutions for data cataloging were created over the last decade so it's getting you know more and more complicated to accelerate those workflows while remaining secure of course hi everybody we are super excited to be uh with matate the like datab

0:22brick C with us uh Matty thank you so much for taking the time today thanks for having me here excited to chat yeah you so you work on so many successful project at dat bricks uh even before dat bricks you worked with mesos uh then you did Apache spark ml flow Unity catalog you work on gen also what what's your focus in these days yeah I'm kind of splitting my time between two things

0:44right now so first I'm still spending a lot of time on Unity catalog um and governance and data sharing um you know there's a lot that we're building out there and it's very important uh to customers to get this right uh and I am also spending some time on gen especially the research part of it figuring out how to make models with better capabilities for what Enterprises

1:07want all right wow awesome okay so M today we like to focus on governance within the air bricks uh this is becoming sh such a big Topic in 2024 with lot of customers uh I I remember a few years ago their RS was uh the spark plus ml company we were doing unifying in know data plus Ai and then came up the lake housee and then now we have the

1:29modern data intelligence platform so I was curious to see you know what's the story behind it around governance what was the vision back then when you when you started their bricks uh you know was it something you always had in mind and wanted to do or is it something new that you see based on your customer needs yeah good question yeah I mean we always um thought when we started that we could build a a complete and modernized data

1:51platform for the things that were changing in the world which is the move to cloud computing where like all your data can actually be you know be centralized in the same Data Center and you can combine all of it in different ways uh and then also the rise of uh machine learning and data science so we always wanted to do that but I think at the at the beginning you know when we were a small company um we had to

2:15focus on what are some use cases what are some things you can do extremely well that existing data tools can't like you can't build the whole platform uh from day one um so there was more of a focus at the beginning of just like working with whatever data people already have rather than us you know being the primary place where you put in all your data and organize it um but one of the things that we did well um what

2:39that and we still do really well um is large scale ETL and data transformation so we very soon saw all those workloads that are loading the data and then it was natural to also build up the Lakehouse part for storage and management and governance um the other thing with governance is that the requirements have changed um and it's getting you know more and more

3:02complicated because there are all these um you know sort of new regulations about data privacy best practices people are figuring out and new use cases such as AI um so we also realized over time that we need to build a more um sophisticated um solution that can support sophisticated policies and that's what we have with unity catalog and governance Is Not A New Concept

3:26multiple solutions for data cataloging were created over uh the last decade so how do you define data governance in 2024 yeah great question um so I think um I think there are a few aspects to it um I think one aspect is um is just basically um management and and search of what you have so do you know what

3:49data you have do you know where it came from um because uh you know there are a lot of rules about what types of data you can use um for each purpose and where it has to be located and so on so that's one um is just basically the cataloging piece of figuring out what you have I think a second piece you really need is the ability to implement flexible policies because the you know

4:11the requirements are going to change over time um you may you know basically you you want to make sure that um um you can easily adapt U your use cases to you know to any new policies as um as they come up um and as part of that I think it's also it's very important to have support for fine grain policies like row and column level security view level

4:34security um these are you know these are things that uh traditional like data Lake doesn't have but something like Unity catalog does because it can do finer grain governance Beyond just who has access to this table or or this file um so these I think are kind of the you know the table Stakes um the other things i' look for are um whether the

4:57the you know the governance system you put in place can help people more easily work with data by helping them discover stuff or by making it easy maybe to when a new data set gets ingested for it to be usable right away by people with the you know with the right permissions and that depends a lot on um how easy it is to you know to detect and tag data with

5:19different properties and to set sort of high level policies that apply to um to the whole catalog and that's something that a lot of customers have been asking about they they say that they want thinks to be more automatic all right so search uh the fine policy with ACL and and and making sure the data can easily be yeah and and the search in particular maybe um I didn't say it super clearly

5:42but I think you really need to know the Providence of it it's not just what do I have but where did it come from you know the lineage does it have some constraints like it has to stay in this geography or that geography um and what purposes is it being used for that's another thing that's changed is it's no longer in many settings it's no longer about like oh this user has access to

6:04this data this one doesn't it depends on the purpose you can use this data for fud detection but not for advertising or not for setting someone's loan rate at the bank or whatever um so that's also so basically yeah it's sort of fine grain um provenance and and understanding the uses of it so you're talking with a lot of customers and lot of Enterprise what do you see as the top

6:26challenges they have currently with governance and the Imp could have on the business yeah great question yeah I think the biggest challenge I think in many places is um they know they have a lot of like really valuable uh and useful data in the company um but it's so complex to manage the policies on it to manage access and quality and make sure everything's okay that many people

6:49in the company don't actually get to use it um so you built in some sense you build this great like data product already but uh people can't use it so they would love to figure out ways to um you know to accelerate those workflows while remaining secure of course um and there are lots of ideas there everywhere from you know automatic uh tagging or lineage uh tracking that

7:11can help you you know more quickly classify data to things like request for Access workflows or um you know masking of data so you see all the non-sensitive parts and then later you can ask for access to the sensitive parts so there are many many ways to address this but this is what we see what is changing now specifically Ally uh like do you see any new uh requirements based or challenges

7:34around gen integration yeah great question yeah yeah definitely um you know gen is changing things up um basically companies are sitting on this this really interesting um text Data it could be like you know human generated stuff like like comments or feedback from people but it could even be machine generated things like logs and um and

7:58they're trying to figure out how to how to best use this and there are some unique challenges with Gen one of the challenges is um you know if you're if you're going to do model training the model might essentially memorize some of the data you can't really tell so like if you train it on data that has say someone's home address or something you might be able to then ask the model and and get that pii out of it so there's a

8:21lot of interest in masking um redacting at the text level before you you train on it to find sensitive stuff there's interest in using AI to analyze uh Text data so you've got a big Text data set with like millions of records and you want to ask you know which of them which of them look like spam you know so I shouldn't include them or which of them

8:44have someone's address can you recognize it um so there's a lot of interest in actually using gen to clean up data to then feed into gen apps whether it's through model training or retrieval uh both cases you need to clean it up um at this datab BX one of the things we've built for that is the AI functions in SQL which let you call a language model

9:06from SQL efficiently on on a you know batch data set to do things like filter records and we're also building a whole family of um kind of standard function for common things like um entity extraction or sentiment analysis and so on where we'll maintain them we'll create the pums and the models and you just call them um and we think that will help a lot with this so um yeah I'd say

9:30I'd say these are the new things beyond all the traditional things like purpose of use you know um geographic location and so on uh is this thing about you know when you use the data Maybe whether you're using it in rag or in model training um you want to make sure that the stuff the model is seeing is you know doesn't um isn't restricted in some way but you still wanted to learn from

9:53it and so so what about using gen directly in the engine itself like there Rick's made a major acquisition last year we ml how is it going to be infused within the governance and unity catalog yeah great question yeah there are a few pieces so so one of them is automatic data tagging um so we're doing more and more to help people tag data and and set policies on it as it comes in uh

10:16based on analyzing it and I think this is going to you know become one of these sets of um of built-in um AI functions that can Tex stuff where they will keep getting better over time um another interesting piece that I think is helping every user already is uh basically the assistance in the UI so everyone's use assistant for um you

10:39know for writing code and and writing SQL or for debugging code but you could also use it to help you understand and debar uh policies for governance I think that would be really interesting um there are already some automated tools that can track you know like what's what are all the you know Downstream assets created from this or what are all the columns D from this column and can show you that on a picture but if you had um

11:03you know um basically a model you could um you could chat with that you can then ask questions about it or tell it okay what do I have to uh change to get this outcome I think it would help um so yeah these are these are the two big ones with governance specifically yeah and because we so we see a lot of players out there and they are talking about governance and data and maybi and gen why do you think theb is so uniquely

11:27positioned to overcome all the Gen challenges and governance yeah I mean I think ultimately um I think companies and users want to unified platform to do this stuff and I think data bakes bet from the beginning of the company that data and AI workloads would be you know would become more more and more unified that it's actually possible to create

11:49like a single engine uh that can handle all of them a single data store you know which is Delta Lake um and a single governance model um that spans them so I think really the Simplicity of the unified platform or like you can set you know policies on some Source data um you can track what it's propagating into and then you can affect everything Downstream including AI applications

12:13like you know in the product or even AI applications that like talk to it to serve data uh that's the thing that's really going to be powerful and then I think another aspect of gen that's really interesting is um as we see you know basically as you see people use the data through many applications again on the same platform you can actually see like hey this data is being used for

12:35machine learning here um it's being used for like you know uh fod detection streaming use cases there and so on you also have an opportunity to learn more about it and to uh make it better for those use cases like to um you know either to say wait this is actually being used in the wrong way or to make things like search and the product better or to make the table um layout

12:59better for those use cases or to do caches and indexes on top um so I think this is another way that AI will help uh AI itself benefits from the unified platform basically but yeah I do think the big one when we talk to companies they just say look I just want an easy way to track stuff across everything and we can offer that and in fact with with um query Federation that we launched

13:22last year we can even offer it for other data stores you can set those up in datab brakes as external data stores and you can get this you know sort of single place to work with everything yeah yeah and and then it's quite amazing to see like you would see the model which is being train and up to and you have all the chains so the data is coming from the source you do the transformation you train that like that and then you have the model deployed Y and then you get

13:44the data back from it and then you can like you can join it with you know like for example you had these chats with the user but then did the user turn or something that's a different table wouldn't it be nice if they were in the same platform so you could just join them so that's that's exactly what we think we do think Ai and data work you know extremely closely together and um you know everyone I work a lot with the

14:07AI research team here and you know I would say like more than half of what they do is looking at data sets and you know creating data sets looking at them figuring out how did this affect this outcome so um I think everyone else will do that too and I believe that was a great introduction because now we're talking about the data intelligence platform and can you tell us more about it um what this is mean yeah for sure so

14:31basically with with generative AI we think that um you know the data platform the concept of a data platform is going to change and in particular and that's why we've actually sort of branded the company around data intelligence platform which is this sort of bigger um uh you know bigger term above Lakehouse where you know Lakehouse is one technology used within it but

14:54data intelligence platform also adds um you know a number of AI Technologies um the biggest thing what does data intelligence mean right is it like just data and intelligence put together no it's it's actually what we mean is that um the platform can learn the semantics of your data based on the contents of it and how it's being used and then it can make it easier for you to work with that

15:17data so for example in many companies the one of the biggest problem that internal users have is they have a question they don't even know what data what tables to like look at to ask about it and then they don't know how to pose that question to the system and get back a reliable answer that's like regardless of how fast your data platform is regardless of how scalable it is um

15:40regardless of how nice the UI looks you know you you still have that problem and with um generative AI we finally have a chance to sort of understand how this what how the data is being used and what it means and to assist users with getting to the right answers quickly same thing with debugging errors same thing with preventing Errors By telling you you know you're you're doing

16:02something different from everyone else who tried to compute this metric um and so on and then not only can we do that for like the data platform users but when you build applications on top if you're building your own internal gen app that thing is going to need to understand the semantics of your data too to say things about it and we can feed that um you know everything we

16:24learned into that so that it can ask questions um so that's in a nutshell that's that's what data intelligence is it's about the data platform actually understanding your data from seeing both its content and how it's how it's used um and we think everyone's going to need this because it's such a huge problem with data platforms alone that like you got all the stuff in there but people don't know how to use it so you

16:47mentioned gen to help uh around governance and for example searching data and finding assets what about the rest of the platform how how do you think it's going to help all Databricks users and I know we have data scientists but also data engineers and even SQL analysts running workows on the air bricks how is Gen going to help them yeah I so I think in in datab briak I think a lot of things will just get

17:09easier like anytime for example one of the main use cases of AI assistance now is anytime you run into some error there's this button that comes up if you have ai features turned on that says diagnose error and it can give you you know really good hints to fix the problem or in many cases it will just fix it for you um and it will it will tell you want to run so that's huge that saves a lot of time you know you can do

17:32this in interactive use and the SQL editor or notebook um you you can also Imagine doing this you know when your production job fails and you know you have to wake up in the middle of the night to you know to fix it um it would be nice to accelerate this process um so I think this is one piece um uh yeah search and Discovery is another piece and I think over time we're going to get

17:55to more and more sort of low code interfaces like you'll be able to build a dashboard just by typing some stuff in fact we already have uh something like that in the new Lake View dashboards um you know you'll be able to sh share it with your friend and help them ask certain questions about it um just in natural language so we want to enable um you know data teams to build that kind of application too in your opinion what

18:19will be a data engineer or data scientist job like in few years are they going to disappear no I think the jobs will still be around uh it's a it's a great question because everyone looks at this code Generation stuff and thinks wow it's so good and then like with data intelligence what if um you know it already knows what all the data where all it is and and what it means and so on but I think they'll still be around

18:42but they'll be working on um somewhat harder problems harder conceptual and business problems and one way I think to convince yourself of this is there are some questions some data science or data questions that you can't answer like say unless you talk to the CFO of the company um you know like for example if I ask something about like some kind of

19:04you know CH metric or something like that you need to figure out what do they you know what do they consider you can't answer the question unless you talk to like the U you know the head of legal or unless you know something about the strategy or whatever so there are some basically you know there are some questions that are just very hard and you need to create a strategy you need to you need to do something about it um

19:27you know even something as simple as like we launched a new product we want to know how it's doing unless you know a lot about the business and the goals for that product it's hard to tell you know what if people bought it but then they're not using it after and they're going to drop what if like you know actually it was free and everyone should be using it but they they aren't because they don't know about it you you just can't answer that by looking at the data

19:49alone so I think data scientists will still be needed to connect the business questions um to uh you know to meaningful analysis uh but their job will just be you know much less sort of you know in the weeds coding stuff and more hopefully more fun you know actually answering the questions yeah they will be able to do bunch more all right awesome thank you so much for time

20:12M we have one last question uh so we need to reach 3,000 subscriber in the YouTube channel uh that's an important milestone for us can you ask the viewer to subscribe to the YouTube channel yeah please like And subscribe it's a great Channel really awesome content here thank you so much thank you so much my