Unity Catalog Governance Strategy & Enterprise Data Mesh

0:05hey hello I hope you're all doing well today I have a special guest as you all know all my guests are special and today I have fabis sign with me hey fabis hi can you introduce yourself our community uh my name is fabis 30 years

0:27old I live in Belgium near gen um and I'm working as a data architect at seika okay can you tell us more about this your company our company yeah sure uh it's not a new company it's founded in 1992 if I'm not mistaken uh it's grown to 9,000 people

0:51already um so in a span of 30 years 9,000 people a global it solution provider um where I'm working at the Data Solutions Department in Belgium um our business unit stems from 2016 so not even 10 years old in Belgium we have around 200 data people and

1:15globally 350 so that that's about the range of our of our company and number of employees um and yeah we're we're a Databricks partner so we are heavily invested in in everything data breaks as well so and I've heard that you're a data breaks Champion so can you tell uh

1:38to our community how did you achieve it so they can maybe do the same thing to apply and become a data break Champion to achieve it well first of all I'm I'm I was the first at zika so I had a a bit different path than if you would be a second champion and so on in in inside of a company or in inside of a

2:01partner um how did I get there well it's mainly doing the right things in your career I guess doing work with spark doing work with Databricks knowing what you're doing in in data engineering or or data in general let's say um in my seven years of experience I had to I was

2:23able to create an to help with building an iot Cloud um and the Azure stack and we build our own data Lake on ad ADLs let's say without Databricks originally so it was only spark 3.0 dockerized container so a bit of technical

2:45baggage um and then yeah transitioned into into Databricks because it uh leverages a lot of things you you don't need to worry about configuring your your spark cluster let's say um and yeah learning all the different Technologies and then I I stepped over to consultancy where I had a lot of different contexts instead instead of

3:08the one iot context and it really opened up a lot of use cases to use those features of Databricks let's say and yeah learning more about the platform helped me become a champion doing your certifications and having a lot of different uh contexts and use

3:30cases uh and having a great Community around you already uh really helps to getting there I guess you started with the um the associate data engineering then the professional one yeah well I did the engineering track um you can you can become a champion either way if you're going data

3:53engineering or because you're a generative AI specialist working on data breaks but mainly it it all boils down to getting um up to a solution architect level knowing what the use cases are that data breaks can help you with knowing the product in and out um and solving cases that are not your

4:15expertise right for example AI is not my expertise I have colleagues who are expert experts on that luckily uh so I took the data engineering part First Data engineering associate certification um I was already quite ready I did the um the partner Academy learning path for it but not not that

4:38much new things uh if you have a few years of experience the professional was a bit trickier yes yeah that then it's more like okay how to do some observability observability monitoring and so on so yeah you have to know hey had to have some bad experience

5:00already to know those things or or at least uh study for them um and yeah after that professional uh I got the opportunity um of our company to go inside of the championship program um I think the next step is following uh a solution architect course from datab bricks so following a

5:24technical uh background at Databricks on on how the solution looks like under the and how to help people achieve their goals um and then to have a panel review to with data breaks themselves um to see if you're kind of fit to follow uh next to the solution architects of Databricks to help them uh solve the issues

5:48at their clients let's say that was very informative well it was a more detailed explanation yeah that's true I think it is worth uh having look at datab bras Comm like data Brak oh gosh I forgot my uh I forgot the

6:10word data breakes platform in order to uh train like take the training the associate data engineering and the professional data engineering then take the course and why not uh get him certified yeah exactly so first of all the training on the on the Academy is amazing uh and if you then can speack it

6:33with real life use cases or even DB demos because that's also a nice uh learning material because it just gives you a full uh Lakehouse experience let's say with some basic notebooks to just get get more of an idea on how to fit everything together from those learning pads that's also for your information fabis I since you're using DB demos or

6:58demo Center now all the um use cases or the examples have been moved to support Unity catalog and all the DBC dashboards now the they support Lake View or that's what we call aibi dashboards so we did a complete migration to support UC and the

7:20new experience all right so they're repated already nice yeah yeah and tell me like uh how have how did you do to build a community inside your company well first of all we're a consultancy company um so every new technology is embraced um and there already was a

7:44community right I I was not the first one at the company we were already using Databricks from our start from the start where we where we were introduced to dat bricks uh I already had quite a few experienced colleagues I had to say um but how do you build out a community that's that's by open by being open sharing things uh being eager having

8:09people eager to learn especially and it's more um yeah a normal process of growth right first hurdles are hard getting people um engaged um doing those no knowledge sharing Brown Bag sessions internal teams discussions blogs sitting

8:32together next to the coffee machine and having a battle on what feature to use on which use case let's say getting people involved that was that was the biggest step um and then finding a nice um organizational pattern to go with it because yeah on once you're having uh thought leaders once you're having amazing Engineers experts on specific

8:56use cases or topics um you still need the process to to get them in line with each other and to not reinvent a wheel every time over and over again find a nice central place where you can put your demos where you can put um your documentation even those stupid things um just centralizing

9:19things getting an idea on uh how to share the information on a structured way uh those are then the next steps and then it's just hoping that everyone inside of your community over or inside your um company will follow you and uh the team let's say um will build it out itself yeah I was I was thinking like

9:42like when you have the data governance here we need to call it let's say people or Knowledge Management how you can structure all the information you do have how you can structure the people or your colleagues to uh let's say to um to create a Unity and then like this this this possibility to share knowledge and because I'm pretty sure you face some

10:05problems that I have never faced and the opposite also true you can exchange experience maybe and then move forward quickly if you face the same problems and I think this is this is something very very interesting and this is something I do in a weekly basis with all my colleagues like we discuss okay tell us something crunchy what happened how you did how were you able to fix this problem and it's like you know

10:28creating your own knowledge base yeah exactly it's it's not it's creating your own knowledge base and the the nice thing about just sharing or listening to other people is that you're getting experienced with sharing yourself and I'm also a team meet at our at our consultancy firm and I'm a team

10:53lead of about eight engineers and now I started in my onet to ones that I have regularly about every two weeks about I also always have the topic um share something you're proud of or what you implemented the last two weeks it just make it can be small wins let's say but they just get aware of

11:17sharing things uh I get to learn things instead of just having to listen or tell them what to do let's say um so it's a win for everyone and it it opens up the sharing Community let's say and if there's a nice thing inside of it that I know okay I'm not the only one who's interested in what you're saying then I just tell them okay put it on our

11:40internal channels uh let everyone know what amazing things you're doing um yeah yeah it's also about recognizing what people what everyone is doing exactly putting this person on the spotlight exactly every everyone at Le even if it's a stupid thing that is already been done still you get the recognition of

12:04you're doing the right thing um and sometimes a lot of times someone thinks wow it was not that hard I just looked at documentation and I found my way around it it can still be a breakthrough for for anyone else and getting the recognition of hey you did something

12:27nice is really really important for for everyone um the as small and our documentation is being updated continuously with new features with new I know only the documentation right uh data breaks is a Never Ending Story of features being introduced or

12:49upgraded um so you're never done learning especially in the data braks field yeah and since uh you mentioned features uh can you just tell us a bit about the exciting use cases you are working on right now well right now I'm more transitioning in growing our community and Consulting the Consultants I call it um because yeah we already

13:14have quite a few experts and uh they're busy on their projects and uh we projects um and the last um use cases that I worked on are mainly where where we helped out design our own best practices um where we uh for example did

13:36um a data warehouse migration from a really nice implemented data warehouse uh Solution on Azure towards a Lakehouse methodology on Databricks Azure Databricks uh at the Technical University of an Hoven um I also helped design a data mesh topology in in um a healthcare

14:01Consortium where they wanted to share um medical research information across different entities uh to get better results in their research um context um what did I guess you you relied on Unity catalog and sharing yeah sure um well in that's more my expertise

14:28so I'm I'm also a Unity catalog subject matter expert so I'm sitting together with um other Champions and people from uh datab brecks to see yeah what what best practices can be distilled um how do we tackle the tougher problems if there is a problem through it inside of a group of peers and try to solve it

14:51um and that that's my focus inside of siika as well um being a Unity catalog expert and helping uh either companies make the right data architecture decisions concerning Unity catalog or how can Unity catalog help them inside of their data governance issues that they currently have let's

15:16say and since you mentioned Unity catalog how hard is it to convince C levels with the U the importance of this subject how hard it is to convince uh people first um so we as a consultancy company are mainly um Project based consultancy uh

15:40which means there is already a use case uh whether it's built me a full data platform then it's quite easy to sell Unity catalog if the use case is okay I have I want to have an AI model um that does X Y and Z based on that specific data set um first of all we need to sell them a

16:02data platform like architecture because uh building a single use case is not the way forward you need to build nice fundaments for it and there is where the fundaments shine um if you're talking about storing data in inside of a data lake or um in general managing multiple data sets um that you are storing in a

16:25data Lake in fact always uh nowadays at least Unity catalog shines right and then it's just easy to go for Databricks as your data platform and use Unity catalog on top um what which is quite easy to sell Unity catalog because now it's quite out of the box with Databricks so we just

16:49have to tell them how amazing Databricks is and that's it um the toughest one to convince uh to go for Unity catalog is for existing Databricks users where you have hi meta stores Etc exactly then the tough part happens um but why would we invest in the refactoring going to Unity catalog

17:12because now our business value is already solved let's say and then yeah one of the biggest things there is getting the right people at the table right you you need to find the correct sponsors first because when you're in a project mode um you're having the use case people next to you or the project managers next to you um

17:37and they might feel the benefit but you you need to find a budget yeah find a budget for those benefits and then you can go for the um an approach where you say okay these are the nice added values if you go for Unity catalog for example yeah governance around your machine learning models uh having full lineage and

18:00especially the lineage is something that sells but you can also go the other way around um do they have a nice solu uh nice answer to some some compliancy um regulations can they show us where their data comes from already um how do they manage um how do they manage management in

18:24general across their in different environments or across their different use cases because a hi metast store typically it it is tied to one having one single workspace but you do not having an encapsulating overview of what is your data estate let's say and that's something Unity catalog nicely solves if

18:46you're already a datab bricks customer and then it's it's fairly easy to sell if it's a client that is looking forward if it's a client that has any aspirations in the AI field then Unity catalog is just an intermediate step of going towards those use case and I'm gonna ask you a tough

19:10question but you know uh I tried in the past some to convince some customers to migrate trity catalog and I faced um sometimes a problem you make you make may call it random problem but that was mainly with asure customers because uh we could like if you want to access the account console you need to have at

19:32Le the global admin or the ad on Asia yeah and this is hard hard to this permission is hard to get even for five minutes yeah um well we're a Microsoft partner and data breakes partner and we are only implementing datab bricks for now on Azure so I can feel your struggle

19:58I um that is true uh sometimes we're in luck that our own development team gets the the privileged identity management of doing these things um where they can just request that specific set because we're quite a big company so we do the full IT

20:20solutions and typically our uh customers are also in in it with AA for um our full Cloud solution let's say um so that's something for those type of customers we can already do but yeah if if you're not involved in that process it's quite hard and it all stems down if

20:44your sponsors is okay with it and it's a a nice validated project at the business already then it's fairly easy then it's just going to it and saying hey this is needs to happen it's in uh it's already aligned with this this and this business person um just do it for us here's the goto guide um and

21:10click on my name make me account admin for the time being or at least um educate the people who need to be account admin from their side but that that's that's only the first step youf right because yeah then you opened up the account console and then the real uh Magic needs to happen um the over overarching account

21:33management and metast store management um is the next step that you need to need to educate everyone in um and it it it has its hurdles right especially on Azure with their SK with the skim provisioning that is not optimal uh yet because there are quite quite a few improvements coming that I heard of at

21:55data n Su so that's nice um so yeah it keeps us busy right yeah exactly and since again you talked about t catalog I have I've heard or I've seen so many customers like looking for the best way or the best strategy to start

22:18the migration from HMS to UC so according to you what's the best strategy like do you advise them to start uh small and then like move let's say project by project or doing the assessment using ucx or well what are your thoughts yeah it's it it all

22:40depends on the client and their current architecture and the size um you can use ucx for doing the full overhaul or or or Project based let's say um but in general it still stems down to who's your sponsor for doing this if it's an enterprise-wide action then definitely

23:03do it in a full overhaul let pick a a beta project let's say to try it out um but do it at once because otherwise there will still be Legacy and in the long run it doesn't help your case um but one of the nice things that I would say

23:25is validate with business first if um on on how to tackle it and do a nice thorough analysis before actually doing the migration and there's where ucx might shine because you can do kind of dry runs let's say and and uh show you quite a few things

23:48um but prior to migrating the first step is indeed already getting that account activated and setting up meta stores for the new projects uh because it's not only current business that needs to um to change but also the new use cases and it might be interesting to wait up to having a new use case on booring or a

24:13new uh project team on boarding let's say where you have new workspaces so they can you can split out the creation of um a Unity catalog environment uh next to the the migration of others um because otherwise you're you might have some impact on on

24:36current processes there it's never an exact use case uh an exact um yeah use case that's happening right uh it can be tricky um but if you ask me go for for a full-fledged implementation from the get-go uh and don't bother about having met a high metast stores anymore

25:00um just to make sure that you have the full encompassing overview and power of unity there yeah indeed also with all the benefits that Unity is bringing like beside the fact that all those system tables that are now available possibility to track the cost the opening all the Gen capabilities lineage

25:23debugging experience I mean that are like almost I can say say like maybe 80% of the features they are now exact on only on UC every feature that um that we now use to sell Databricks let's say is built on top of unity and mainly about around those

25:45system tables right if if someone is talking about okay I can see data lineage yeah fine it's due to those system tables or um I I had a nice question last week about um how can I do auditing on who accessed what data for me it was quite simple there are audit tables inside of unity catalog

26:07where you can find all every action that is happening every credential vending that happened for a specific user is registered already inside of an audit table it's to a year there there was a requirement of having it 10 years let's say yeah it's fairly easy just have a standard ETL pipeline for you

26:29reading out that table and storing it even longer um but yeah it it stores a lot of information um and I'm really hoping up to the point where I don't have to build my Lakehouse views or my AI bi dashboards um on top of those system tables I'm I I'm hoping for uh The Lakehouse observability project inside

26:52of Databricks to give me uh nice nice dashboards out of the box coming with with Databricks um and and to come back on on the hive metast store thing that's exactly where where Unity catalog can shine and having um governance workspaces one encompassing governance

27:14workspace where you can monitor everything that is happening inside of your Databricks account instead of having to hope that every workspace does the right thing where now with system tables you can just watch what's happening in every workspace and uh you know uh what I did

27:36I think just this morning I was using Jenny spaces with system tables to ask some random questions hey tell me which tables uh which table has been most accessed uh recently in this month for example and then uh who accessed this specific table or who modified and who can access this one and jnie was able to

27:59answer those questions uh instantly without knowing any SQL code which means anyone can just go and ask some random questions over system tables and then of course it won't give response every time but you can give more context add your instructions and you can uh be ready to go exactly the nice thing about the system tables is that all the metadata

28:22around those tables is already filled in if you would do the same thing about your own data table Genie is also able to do it on your own data right um so it's it's always nice to see that um everything that is locked inside of Databricks can be reused to find new

28:45business value or at least Sol solve some uh some questions that business might have um yeah and I have also a question for you like I this is something I always heard hear from customers mhm how uh should I implement the catalog schema and table level like should should the catalog be production

29:07I have like one catalog for production one catalog for staging one for div or should I have different cataloges and in every single catalog a schema with production Dev staging I think this is the question I hear the most yeah it's it's it's all based on semantics and on how you you you build your um infrastructure

29:30typically um if you're having one workspace where you're doing your development and your production is run workflows are running on the same thing for me it doesn't matter but what my so so these are our our best practices as a consultancy company that I'm giving away freely now or at least it's it's my interpretation of them

29:54thank you for what what what I would suggest is um so you're you're having the three level Nam space catalog schema and uh table name for example um on a catalog level I use a three L name space as well being the entity where you're running

30:16for for example Finance then the project name of that entity and then the environment that's my typical goto catalog name that enables a data mesh architecture by having different different business units to have their own Solution on a product level you also

30:38are able to do it and then have a catalog per every environment that's my my go to case why because then you can um have a default location on your catalog because now without the meta store storage which I would advise not

31:00to use um you can say okay for this Catalog Store my data in this specific storage container bucket storage count um which is physically separate from another one so then your production environment is safe from your development environment and it's based on a catalog

31:22level if then a non-knowing user creates a random schema inside of that catalog every data is still stored at that specific storage location right so production data is always separated from EnV development data that's the main goal of putting production and development in separate

31:46cataloges for me because schemas are managed by users and not by System admins let's say or infrastructure people um and it guarantees that your data is secured in the same fashion as the data that it's collocated with um so that's my advice on the development and production uh catalog

32:11suggestion can do the same thing Ona level right but it makes it a bit trickier uh to do it um yeah got you got you thank you and by the way for those who are watching this they can still benefit from the catalog segregation which means this catalog can only be accessible from the production it can only be like can read

32:33and write from the production workspace it can only be read from another for example catalog if they want to do some testing another another EAS easy thing is if you're having the scoped at the um business entity project and environment level typically it will stem together with the specific workspace you're using

32:57and you can put the default catalog of that workspace up to that catalog and 95% of the time you don't need to worry about the three-level namespace anymore but you can still use a two-level nam space as well so it it's a nice tip to help um especially for uh less experienced users

33:19or less technical users to Encompass only the the data that they need let's say um makes it easier to adopt and I have one more question for you before sure we close if you have to choose only one

33:41feature from data breaks what would it be and of course Unity catalog is not an answer because it covers it's a wrapper over tons of features so Unity catalog does not count breaks itself right tough one um I think something something something

34:03maybe you wanted to have when you were like data engineer at the beginning of your career uh well first of all the management of clusters it's it's underrated still um uh I don't know a lot of Engineers who can uh manage that much clusters uh inside of it um it's maybe not a feature but in

34:27general the developer experience I think I can say terraform module and dat recessive bundles I think those are really beneficial things of having cicd up and running um and to me it feels every project uh stays ahead of their time or not

34:49based on how um how their cicd and devops uh processes are implemented if if you can deploy easily to a new environment or you can deploy easily to your development and your production environment then you're already set for value and then it's just finding the business context and and

35:12getting there um and I think those are the the best value that Databricks offers it's just the developer experience is that um too broad of a feature or are you hoping for another answer no I think you you got it right um like the dev experience is something that I hear

35:36a lot uh from customers because they're a they're able to use notebooks they're able to use uh IDs if they want they have data breaks connect now they did integrate with DB connect with serverless offering uh we have the assistant that can help them you know uh debug their their problem the autoc completion now like and now was just writing query this

35:59morning I was querying system tables and I was like okay I was looking for I think requestor params and then it suggest yeah you should to take the email and then I add end and it automatically prompt request params and then added the um cluster ID and this is yeah it's not gamechanging but it just improve or my productivity and I don't

36:22need to go to do some back and forth to get the schema name or the missing column and this is something important like because if you're able to improve the productivity I don't know by just 30 minutes a day it's at at the end of the month it's at least I don't know uh 10 hours so it's literally you're winning one one one day per per month and it's 12 days per per year I mean it's it's a good uh

36:47Improvement and now that you're talking about productivity that's that's a nice thing to get because from uh new data Engineers or even experienced data engine is I also get often get the question yeah what uh node type should I use for my clusters or or how should I scale my clusters and then my typical answer is first identify if you need to

37:10tweak it uh because you're the most expensive one it's not the cluster that is the expensive one um it's the expert that is developing that's that that's the most costly one it's always better to um first identify if it's needed if there is potentially a need

37:33try fton is my first answer um and otherwise I'm I'm hoping in the near future that uh the server this domain which Databricks is Heavenly investing in um gets the T-shirt size option as that data briak SQL has um because you're always aware of how big

37:56of size the load will be um but not having to worry about configuring every knob uh would be great but um for me currently the date brick equals serverless uh the DAT brick serverless uh workflows for example need at least one or two knobs um to work with uh even if it's

38:21just for having the technical experts uh being happy that they can do some Transformations on that cluster um but in general the way where Databricks is going in in managing things for you it just helps you solve the productivity issue um yeah but it should with the

38:44serverless yeah especially with serverless right it it it's really helping but I'm just missing that tiny knob of having the T-shirt sizing of where serverless is going which your uh with your costs because every business productivity is a cost game and you're just transitioning it from your um uh

39:06the classic offering yeah from from the classic offering where uh the developer is your expensive cost towards uh infrastructure being your expensive cost um which can already be solved by just limiting with the T-shirt size offering let's say but I'm fairly certain in the near future datab bricks will come with with an answer on top of that as well so

39:30that that's for sure that's for sure thank you thank you fabis for uh your time thanks for sharing uh your uh your experience your wi knowledge about the platform at also your expertise um I was lucky to have you um with me in this uh during this session

39:52thank you again fais thank you bye bye