Executive Interview
The future of Governance and AI with Databricks CTO Matei Zaharia
Matei Zaharia · Co-founder & CTO · LinkedIn
things that were changing in the world the move to cloud computing all right so search sh policy with ACL and making sure the data can easily be leverage governance Is Not A New Concept multiple
solutions for data cataloging were created over the last decade so it's getting you know more and more complicated to accelerate those workflows while remaining secure of course hi everybody we are super excited
to be uh with matate the the like datab brick C with us uh Matty thank you so much for taking the time today thanks for having me here excited to chat yeah you so you work on so many successful
project at dat bricks uh even before dat bricks you worked with mesos uh then you did Apache spark ml flow Unity catalog you work on gen also what what's your focus in these days yeah I'm kind of
splitting my time between two things right now so first I'm still spending a lot of time on Unity catalog um and governance and data sharing um you know there's a lot that we're building out
there and it's very important uh to customers to get this right uh and I am also spending some time on gen especially the research part of it figuring out how to make models with better capabilities for what Enterprises
want all right wow awesome okay so M today we like to focus on governance within the air bricks uh this is becoming sh such a big Topic in 2024 with lot of customers uh I I remember a
few years ago their RS was uh the spark plus ml company we were doing unifying in know data plus Ai and then came up the lake housee and then now we have the modern data intelligence platform so I
was curious to see you know what's the story behind it around governance what was the vision back then when you when you started their bricks uh you know was it something you always had in mind and
wanted to do or is it something new that you see based on your customer needs yeah good question yeah I mean we always um thought when we started that we could build a a complete and modernized data
platform for the things that were changing in the world which is the move to cloud computing where like all your data can actually be you know be centralized in the same Data Center and
you can combine all of it in different ways uh and then also the rise of uh machine learning and data science so we always wanted to do that but I think at the at the beginning you know when we
were a small company um we we had to focus on what are some use cases what are some things you can do extremely well that existing data tools can't like you can't build the whole platform uh
from day one um so there was more of a focus at the beginning of just like working with whatever data people already have rather than us you know being the primary place where you put in
all your data and organize it um but one of the things that we did well um what that and we still do really well um is large scale ETL and data transformation so we very soon saw all those workloads
that are loading the data and then it was natural to also build up the Lakehouse part for storage and management and governance um the other thing with governance is that the requirements have changed um and it's
getting you know more and more complicated because there are all these um you know sort of new regulations about data privacy best practices people are figuring out and new use cases such
as AI um so we also realized over time that we need to build a more um sophisticated um solution that can support sophisticated policies and that's what we have with unity catalog and governance Is Not A New Concept
multiple solutions for data cataloging were created over uh the last decade so how do you define data governance in 2024 yeah great question um so I think um I think there are a few aspects to it
um I think one aspect is um is just basically um management and and search of what you have so do you know what data you have do you know where it came from um because uh you know there are a
lot of rules about what types of data you can use um for each purpose and where it has to be located and so on so that's one um is just basically the cataloging piece of figuring out what
you have I think a second piece you really need is the ability to implement flexible policies because the you know the requirements are going to change over time um you may you know basically
you you want to make sure that um um you can easily adapt U your use cases to you know to any new policies as um as they come up um and as part of that I think it's also it's very important to have
support for fine grain policies like row and column level security view level security um these are you know these are things that uh traditional like data Lake doesn't have but something like
Unity catalog does because it can do finer grain governance Beyond just who has access to this table or or this file um so these I think are kind of the the you know the table Stakes um the other
things i' look for are um whether the the you know the governance system you put in place can help people more easily work with data by helping them discover stuff or by making it easy maybe to when
a new data set gets ingested for it to be usable right away by people with the you know with the right permissions and that depends a lot on um how easy it is to you know to detect and tag data with
different properties and to set sort of high level policies that apply to um to the whole catalog and that's something that a lot of customers have been asking about they they say that they want
thinks to be more automatic all right so search uh the fine policy with ACL and and and making sure the data can easily be yeah and and the search in particular maybe um I didn't say it super clearly
but I think you really need to know the Providence of it it's not just what do I have but where did it come from you know the lineage does it have some constraints like it has to stay in this
geography or that geography um and what purposes is it being used for that's another thing that's changed is it's no longer in many settings it's no longer about like oh this user has access to
this data this one doesn't it depends on the purpose you can use this data for fud detection but not for advertising or not for setting someone's loan rate at the bank or whatever um so that's also
so basically yeah it's sort of fine grain um provenance and and understanding the uses of it so you're talking with a lot of customers and lot of Enterprise what do you see as the top
challenges they have currently with governance and the Imp could have on the business yeah great question yeah I think the the biggest challenge I think in many places is um they know they have
a lot of like really valuable uh and useful data in the company um but it's so complex to manage the the policies on it to manage access and quality and make sure everything's okay that many people
in the company don't actually get to use it um so you built in some sense you build this great like data product already but uh people can't use it so they would love to to figure out ways to
um you know to accelerate those workflows while remaining secure of course um and there are lots of ideas there everywhere from you know automatic uh tagging or lineage uh tracking that
can help you you know more quickly classify data to things like request for Access workflows or um you know masking of data so you see all the non-sensitive parts and then later you can ask for
access to the sensitive parts so there are many many ways to address this but this is what we see what is changing now specifically Ally uh like do you see any new uh requirements based or challenges
around gen integration yeah great question yeah yeah definitely um you know gen is is changing things up um basically companies are sitting on this this really interesting um text Data it
could be like you know human generated stuff like like comments or feedback from people but it could even be machine generated things like logs and um and they're trying to figure out how to how
to best use this and there are some unique challenges with Gen one of the challenges is um you know if you're if you're going to do model training the model might essentially memorize some of
the data you can't really tell so like if you train it on data that has say someone's home address or something you might be able to then ask the model and and get that pii out of it so there's a
lot of interest in masking um redacting at the text level before you you train on it to find sensitive stuff there's interest in using AI to analyze uh Text data so you've got a big Text data set
with like millions of records and you want to ask you know which of them which of them look like spam you know so I shouldn't include them or which of them have someone's address can you recognize
it um so there's a lot of interest in actually using gen to clean up data to then feed into gen apps whether it's through model training or retrieval uh both cases you need to clean it up um at
this datab BX one of the things we've built for that is the AI functions in SQL which let you call a language model from SQL efficiently on on a you know batch data set to do things like filter
records and we're also building a whole family of um kind of standard function for common things like um entity extraction or sentiment analysis and so on where we'll maintain them we'll
create the pums and the models and you just call them um and we think that will help a lot with this so um yeah I'd say I'd say these are the new things beyond all the traditional things like purpose
of use you know um geographic location and so on uh is this thing about you know when you use the data Maybe whether you're using it in rag or in model training um you want to make sure that
the stuff the model is seeing is you know doesn't um isn't restricted in some way but you still wanted to learn from it and so so what about using gen directly in the engine itself like there
Rick's made a major acquisition last year we ml how is it going to be infused within the governance and unity catalog yeah great question yeah there are a few pieces so so one of them is automatic
data tagging um so we we're doing more and more to help people tag data and and set policies on it as it comes in uh based on analyzing it and I think this is going to you know become one of these
sets of um of built-in um AI functions that can Tex stuff where they will keep getting better over time um another interesting piece that I think is helping every user already is uh
basically the the assistance in the UI so everyone's use assistant for um you know for writing code and and writing SQL or for debugging code but you could also use it to help you understand and
debar uh policies for governance I think that would be really interesting um there are already some automated tools that can track you know like what's what are all the you know Downstream assets
created from this or what are all the columns D from this column and can show you that on a picture but if you had um you know um basically a model you could um you could chat with that you can then
ask questions about it or tell it okay what do I have to uh change to to get this outcome I think it would help um so yeah these are these are the two big ones with governance specifically yeah
and because we so we see a lot of players out there and they are talking about governance and data and maybi and gen why do you think theb is so uniquely positioned to overcome all the Gen
challenges and governance yeah I mean I think ultimately um I think companies and users want to unified platform to do this stuff and I think data bakes bet from the beginning of the company that
data and AI workloads would be you know would become more more and more unified that it's actually possible to create like a single engine uh that can handle all of them a single data store you know
which is Delta Lake um and a single governance model um that spans them so I think really the Simplicity of the unified platform or like you can set you know policies on some Source data um you
can track what it's propagating into and then you can affect everything Downstream including AI applications like you know in the product or even AI applications that like talk to it to
serve data uh that's the thing that's really going to be powerful and then I think another aspect of gen that's really interesting is um as we see you know basically as you see people use the
data through many applications again on the same platform you can actually see like hey this data is being used for machine learning here um it's being used for like you know uh fod detection
streaming use cases there and so on you also have an opportunity to learn more about it and to uh make it better for those use cases like to um you know either to say wait this is actually
being used in the wrong way or to make things like search and the product better or to make the table um layout better for those use cases or to do caches and indexes on top um so I think
this is another way that AI will help uh AI itself benefits from the unified platform basically but yeah I do think the big one when we talk to companies they just say look I just want an easy
way to track stuff across everything and we can offer that and in fact with with um query Federation that we launched last year we can even offer it for other data stores you can set those up in
datab brakes as external data stores and you can get this you know sort of single place to work with everything yeah yeah and and then it's quite amazing to see like you would see the model which is
being train and up to and you have all the chains so the data is coming from the source you do the transformation you train that like that and then you have the model deployed Y and then you get
the data back from it and then you can like you can join it with you know like for example you had these chats with the user but then did the user turn or something that's a different table
wouldn't it be nice if they were in the same platform so you could just join them so that's that's exactly what we think we do think Ai and data work you know extremely closely together and um
you know everyone I work a lot with the AI research team here and you know I would say like more than half of what they do is looking at data sets and you know creating data sets looking at them
figuring out how did this affect this outcome so um I think everyone else will do that too and I believe that was a great introduction because now we're talking about the data intelligence
platform and can you tell us more about it um what this is mean yeah for sure so basically with with generative AI we we think that um you know the the data platform the concept of a data platform
is going to change and in particular and that's why we we've actually sort of branded the company around data intelligence platform which is this sort of bigger um uh you know bigger term
above Lakehouse where you know Lakehouse is one technology used within it but data intelligence platform also adds um you know a number of AI Technologies um the the biggest thing what does data
intelligence mean right is it like just data and intelligence put together no it's it's actually what we mean is that um the platform can learn the semantics of your data based on the contents of it
and how it's being used and then it can make it easier for you to work with that data so for example in many companies the one of the biggest problem that internal users have is they have a
question they don't even know what data what tables to like look at to ask about it and then they don't know how to pose that question to the system and get back a reliable answer that's like regardless
of how fast your data platform is regardless of how scalable it is um regardless of how nice the UI looks you know you you still have that problem and with um generative AI we finally have a
chance to sort of understand how this what how the data is being used and what it means and to assist users with getting to the right answers quickly same thing with debugging errors same
thing with preventing Errors By telling you you know you're you're doing something different from everyone else who tried to compute this metric um and so on and then not only can we do that
for like the data platform users but when you build applications on top if you're building your own internal gen app that thing is going to need to understand the semantics of your data
too to say things about it and we can feed that um you know everything we learned into that so that it can ask questions um so that's in a nutshell that's that's what data intelligence is
it's about the data platform actually understanding your data from seeing both its content and how it's how it's used um and we think everyone's going to need this because it's such a huge problem
with data platforms alone that like you got all the stuff in there but people don't know how to use it so you mentioned gen to help uh around governance and for example searching data and finding assets what about the
rest of the platform how how do you think it's going to help all data bricks users and I know we have data scientists but also data engineers and even SQL analysts running workows on the air
bricks how is Gen going to help them yeah I so I think in in datab briak I think a lot of things will just get easier like anytime for example one of the main use cases of AI assistance now
is anytime you run into some error there's this button that comes up if you have ai features turned on that says diagnose error and it can give you you know really good hints to fix the
problem or in many cases it will just fix it for you um and it will it will tell you want to run so that's huge that saves a lot of time you know you can do this in interactive use and the SQL
editor or notebook um you you can also Imagine doing this you know when your production job fails and you know you have to wake up in the middle of the night to you know to fix it um it would
be nice to to accelerate this process um so I think this is one piece um uh yeah search and Discovery is another piece and I think over time we're going to get to more and more sort of low code
interfaces like you'll be able to build a dashboard just by typing some stuff in fact we already have uh something like that in the new Lake View dashboards um you know you'll be able to sh share it
with your friend and help them ask certain questions about it um just in natural language so we want to enable um you know data teams to build that kind of application too in your opinion what
will be a data engineer or data scientist job like in few years are they going to disappear no I think the jobs will still be around uh it's a it's a great question because everyone looks at
this code Generation stuff and thinks wow it's so good and then like with data intelligence what if um you know it already knows what all the data where all it is and and what it means and so
on but I think they'll still be around but they'll be working on um somewhat harder problems harder conceptual and business problems and one way I think to convince yourself of this is there are
some questions some data science or data questions that you can't answer like say unless you talk to the CFO of the company um you know like for example if I ask something about like some kind of
you know CH metric or something like that you need to figure out what do they you know what do they consider you can't answer the question unless you talk to like the U you know the head of legal or
unless you know something about the the strategy or whatever so there are some basically you know there are some questions that are just very hard and you need to create a strategy you need
to you need to do something about it um you know even something as simple as like we launched a new product we want to know how it's doing unless you know a lot about the business and the goals for
that product it's hard to tell you know what if people bought it but then they're not using it after and they're going to drop what if like you know actually it was free and everyone should
be using it but they they aren't because they don't know about it you you just can't answer that by looking at the data alone so I think data scientists will still be needed to connect the business
questions um to uh you know to to meaningful analysis uh but their job will just be you know much less sort of you know in the weeds coding stuff and more hopefully more fun you know
actually answering the questions yeah they will be able to do bunch more all right awesome thank you so much for time M we we have one last question uh so we need to reach 3,000 subscriber in the
YouTube channel uh that's an important milestone for us can you ask the viewer to subscribe to the YouTube channel yeah please like And subscribe it's a great Channel really awesome content here
thank you so much thank you so much my