Unity Catalog Migration: Advice from a Databricks MVP

0:07hey F today I have Eddie and one of the data breakes MVPs hi Eddie hi you I'm super happy uh to have you if I'm not wrong you're the fifth MVP um I interview uh this year so tell us more uh about you about

0:33your role like Enlighten us Absol absolutely thank you for having me on the on the show first of all and uh my name is Eddie I'm the CTO at coine Tech we uh this week we it was announced that data R Ventures uh invested in quch so we're we're uh we've been around for about three years and uh during that time we've serviced about 400 customers

0:57and and we're trying to grow like crazy we're uh trying to double our growth and things like that so it's uh but I guess myself personally I started using Databricks back in 2016 prior to that I was a data architect and uh just fell in love with the platform it it solved so many of the problems that I've that I've experienced in my career uh with respect to data warehouses and and data lakes

1:20and things like that and the Lakehouse Paradigm uh is amazing and I I just just fell in love with it and I've wanted to do it ever since and so about 3 years ago AG go uh I was introduced to uh our our CEO at soini and we decided we wanted to build a Databricks exclusive uh practice Solutions integrator practice and that's what we've done so uh so yeah happy to be here and uh yeah

1:43excited excited to be on this journey with you guys so it's amazing you know it's the first time I metet someone who used the um the platform before me because I used it the first time in 2017 uh uh for my uh for my master degree and like and I end up joining the

2:05company a few years later yeah so so I guess you have seen like how the platform evolved like uh um like between like I don't know 2018 and 2024 so what if you try to reflect in the in the past like what would be the

2:27your comment right now like regarding how like it's expanded to cover almost everything yeah absolutely I think uh early on obvious obviously it was it started out as as a spark platform uh and uh I my experience trying to use spark prior to Databricks uh I'm not a Linux guy I'm not uh you know that's not

2:49my my my and being able to manage or having to manage infrastructure and networking and all those complexities uh whether on premises or even in in the cloud it was just hard for me personally and so finding uh you know data brecks being you know the commitment to being a managed spark platform initially was was very compelling to me it helped me get get ac across some of the roadblocks that I had personally and you know being

3:13able to do what I do best on the data side of things and not not have to deal with all the complexities of the of the infrastructure and so forth um and since that time uh you know early on I think my first exposure to Databricks was through uh the moo uh training program that I think there was some early spark 2 training that was that was available and took that and I just it was just so eye openening to me uh and

3:36again I I had been managing uh data Lakes using Hadoop and things like that and the complexities of getting you know essentially having to override data all over and over again this is pre- Delta format things like that um but you know over time Databricks has just surmounted every one of the challenges that I that I've had in in my past data platform experiences and the

3:58two that I think most Monumental uh in in my experience has been Unity catalog and uh investment in the data warehouse and kind of inventing this this Lakehouse uh you know Paradigm uh uh you know taking the best of data warehousing and the best of data lakes and being able to create this unified platform that does everything very very well so uh that's been my experience it's been

4:21it's been amazing so and since like you mentioned the unity catalog I am still wondering like how we couldn't how we didn't think about it like I don't know few years back like we're just I feel like almost all companies like didn't care about maybe security aspects like yeah it was

4:44important but not not as much as now and I was like wondering like like in the past like almost everyone had access to the data Lake almost all of like people I worked with for like uh admin they could do almost everything compared to now where we try to do to give the like the least

5:08privilege possible to make sure and of course make sure that we have this multiactor authentication U keys and SSO and like do a lot of like security awareness just to make sure that nothing nothing leaks and I think on the top of it we have uh this UC with the audit log and system tabl that enables you like

5:32not only to understand who has access to the data but when they did access the uh the data and you can can you tell us more about um I don't know maybe tell us more about the latest project databas project you've worked on like why was it maybe challenging why was it rewarding like en Laten us to know to be on

5:54your shoes in your shoes yeah absolutely as CTO I don't get to be as handson as I used to be you know in my past but uh we at quch we manage a lot of lot of projects uh and I would say that our superpowers if you will are unity catalog upgrades and migrations uh and Lakehouse build outs for for new customers and we're also doing quite a

6:16bit of you know gen AI use cases using Mosaic Ai and things like that but I think uh so more more most recently a lot of a lot of unity catalog upgrades I think a lot of customers are uh you know who have been on Databricks prior uh have needed a little bit of help and assistance getting on to Unity catalog initially and and getting migrated to the UC meta store and so we

6:38we can help with that and we've done quite a bit of quite a few projects uh you know involving that um and the other obviously as Databricks has has gained momentum uh everybody you know was kind of the lak house Paradigm has one over and especially this new data intelligence Paradigm as well and uh everybody wants to move to Databricks and so uh we're trying to help customers

7:00uh overcome those you know the complexities of moving from you know a legacy cloud data warehouse or an on-premises data warehouse to Databricks through uh through you know you leveraging the AI and the capabilities of the Databricks platform so those are some of the projects that we've worked on most most recently are are some of those like Unity catalog and migrations uh quite a few of those

7:23so and and since you mentioned like USC migrations what would be your advice to maybe customers or user who are who are just like who haven't done it or maybe are on the process of the migration what would be your advice like should they like how would they process or start the process because I know sometimes can be little bit long to migrate tables

7:46workflows add maybe the third level name space what would be your advice to them yeah super super question I think uh obviously we kind of want to want to do a little bit of design up front uh you know the the beauty of catalog is enables you to kind of implement your your desire to be State whether that's a data mesh whether you know that's

8:08traditional Medallion zones or uh you know maybe you have different business units in in your company who who want to have isolation by workspace uh I love how Unity catalog you can you can essentially you know basically create a per purpose-built uh secure you know secure environment for uh whatever the customer's needs are you know you can secure at the workspace level you can

8:31secure at the storage you know the storage layer and of course through the identity groups and and things like that um being able to you know or organize it appropriately so I would say and my my advice would be to you know we have automated assessments a lot of Partners have those types of things we can come in quickly assess your environment and then advise on uh what

8:54the 2B State needs to look like U and then it's it's fairly it's not too trivial or not too ult to configure Unity catalog and I think where the work comes in a lot of times is all the Legacy workflows and and pipelines that need to be upgraded and of course tables and the you know the code and and so forth the Clusters and compute and things like that so that's where we can

9:17additionally come in through applying accelerators and using the data RS accelerator such as ucx and others as well yeah USDX has i' I've tested like us X when it was first released it was just handling if I'm not wrong like migrating groups and tables and now like I've seen all the things ucx is capable of I was

9:41just like oh I mean it's really simplifying a lot of a lot of things and I would have loved to have it to have this version like maybe two years ago exactly so yeah it's impressive I I it's the the the velocity with which the the the new features

10:03and and enhancements come out from ucx it's impressive I kudos to that team so yeah I mean because everything now relies on Unity catalog if you go there's like I think all the features now relies on UC like whether it's serverless uh some parts of workflows online store feature store DLT I mean I'm pretty I start thinking

10:28it's like almost mon mandatory to have it yeah you can still work on hi but you won't you will just literally just have spark and manage spark and that's it while UC brings a lot of things and the good thing is that with I don't know if you tried the um the uh you see open source again it's just amazing and I've seen the road map like now you can read

10:52and write from to external UC tables and you can read manage tables and in the next release they plan to add the writing from like uh external uh engines to manage use manage shable it's just amazing which means you can literally do everything like bring d d I

11:14don't know open source park or any other engine supported and you're good to go it's it's incredible yeah I think uh I that's that's U what as you pointed out I think you need to catalog it un the data intelligence platform uh at all the features you know if you want to do AI uh on data breaks you need Unity

11:36catalog and and I think the the thing that it's been amazing to me is that how it open or breaks down all the silos you know silos has kind of been the theme of my past career you know I want to access data in this database or you know this Source system things like that but it's kind of a wall Garden I can't get access to it and through you know even if I don't have don't want to

11:59investing in in just ingesting all that data into Databricks I can I can Federate you know and I can Delta share and so just you know it's a means of opening up the platform and and you know just you know widening the universe of all the things that that are capable to us so I think that's that's that's an amazing feature so and can you also tell us more about

12:22your experience as an MVP but first like how how how did you become an MVP and then talk talk about your experience yeah I think uh you know as CTO I don't get a chance to be quite as Hands-On anymore but I I try to be a thought leader both within my company and and and outside as well um I'm I'm I'm kind of new to the podcasting

12:45and the the you know the web type type type engagements but I've been writing lots of blogs and things like that so that's kind of where I've I've lived and you know obviously doing LinkedIn posts and and so forth but uh yeah I you know quite Frank I think you know the community of MVPs there are some ones out there that are my idols and I you know I don't I don't profess to be at their level at this point but

13:08um you know I'm I I as you can tell I'm very very passionate about the data R platform and uh and I I I just want to be a champion and advise customers on on uh you know getting the most value out of it because it is an amazing platform so and um uh also so I think it's same

13:30question I asked to many many MVPs but what would be your recommendation to someone who want to learn like how to use data braks or like want to you know deep dive on or enhancing the knowledge what would be the steps according to you that's a super question I think uh what's worked for me is uh obviously

13:53being being you know maybe blocking out your calendar if you have the ability to do that for an hour a day to read uh you know catch up on blogs uh you're you know uh NextGen lak house for example that the newsletters that come out every month are amazing and the videos that that that you guys put out uh being you know keeping up on on on that and then

14:16uh putting it into practice is would be my recommendation you know it's it's it's becoming easier uh you know uh to be able to spin up a Databricks environment and to You Know download a set of notebooks and and be able to test things out yourself and and kind of be Hands-On and um the combination of those two things I think is the is key uh and then just doing that on a

14:38regular basis not just once once a week or something like that but daily over time it's going to it's going to pay dividends I think yeah you're you're you're totally right and I think that's how I started like U like developing my data as like reading blog Bust or reading the data breakes Community like it's technical blog Community um technical blog um

15:02reading documentation and of course I think practicing is very important it's good to have a like uh the knowledge but you need to practice it because when you practice then you're going to face I don't know some problems some issues and it will help you understand how the products work and of course since almost like I don't know maybe around 80% of

15:24the tools you're going to be using are open source so I'm pretty sure you will find answers in like like redit or stock overflow or even you by using CH GPT or perlex city or any llm uh you'll be you'll be using I think you have and I still believe that people who are like like maybe graduating RA are so lucky because they have so many tools

15:46they can leverage I mean if I had perplexity or ship when I was doing my master degree I think I would have nailed it I agreed you know just more over like the I assisted development today you know like the D Rick's assistant for example or or you know just using yeah using llms to help you learn things and and to train and to help you even write

16:09code uh so that's my my Saving Grace you know nowadays I don't get a chance to be be Hands-On quite as much but uh D the D R's assistant comes to the rescue you know it helps me remember how to how to do such and such in you know P python or P spark things like that so yeah yeah I mean it's super it's it's super helpful and useful so no longer need to learn by

16:32heart like the API just like write some random stuff and then ask hey can you can you bring me like can you tell me the API should use here yes of course here we go yeah it's incredible and I also have like I think I've also seen more and more uh customers asking questions about the phobs I mean how do you handle those kind of

16:54topics yeah I so going we've we've you obvious that's the another thing that Unity catalog enables is all the this the system tables and the visibility and observability into into the cost things and prior to the system tables there were Databricks audit logs and things like that and one of the common questions that that our some of our customers have always had is uh you know

17:18I i' I've got my data breaks costs and of course but I've also got my underlying hyperscaler uh you know AWS or Azure cost things like that and I want to I want to unify them and of course that's becoming less of an issue nowadays with serverless but uh nevertheless we they they want to see an all of view of that and so uh we' quch We've we've built uh aibi dashboards for

17:40example that will um that will surface all that and kind of create a single painted glass that that uh provides observability and uh and also operational dashboards and things like that so um and I'm not to say not to say that ours are the best but I know that there are a lot of free and and uh you know open source ones out there as well um and lots of lots of other MVPs and

18:02others have have created great observability dashboards so uh pick the best ones that that you have and you know if you want to build your own it's it's entirely possible to do that using the system tables and data breaks so uh that would be my recommendation is and and obviously just keep keep an eye on it and then U you know obviously with that you can you can create alerts you can create controls on on on you know on

18:25your your clusters and things like that and and create apply some governance so yeah and I think I'm going to share with you something I haven't shared before but uh I created this like notebook notebook like uh before system tables I was like creating some tables from system tables so basically I

18:47did enable in the past the verbos audit logs uh in uh in ADLs and then I brought uh autol loader that was like bringing the Json file and then I put a notebook like where I was where I where I was parsing the Json to make sure that I create tables and then I was running those like a workflow that trigger those

19:09like those verbos in a weekly basis and then I was like running a dashboard to see it was not about the cost because back in that time I don't think we had the cost but we had like all the activities like who created The Notebook who created the table and stuff like this and and like few uh I think few months later like system tables were announced and that's it's like okay

19:31let's just throw those notebooks why building things why spending compute to run those things and also make sure that and it took me time because I had to par a lot of a lot of C like um Json make sure to have like the right schema and now just they are just there don't need to pay anything they are just available just go and query them rather than having to par them and now we're also

19:54doing I think more things because we're bringing now like uh like on the shelves dashboard that you can just deploy to track the cost uh like workflows for like we're bringing things for the assistant for Marketplace and going to have bunch like a lot of tables and I'm pretty sure Greg uh from The Money Team will walk us through those amazing uh

20:18features to have this observability and who's better than data breaks to monitor data R cost I guess exactly yeah I think you know to your point like early on you know D rcks had the OverWatch was is a great platform especially if a big customer with lots and lots of workspaces and things like that being able to centralize your costs and and

20:41manage that and I think over time you know I'm seeing that you know some of those OverWatch customers want to want to move to system tables and kind of modernize their approach like you just said so I think that's a yeah common thing and I'm I really love all the latest uh I think I think I've noticed that as well well in the last SE several months as data rck is focusing more on finops and and making sure that data rck

21:05is the most cost-effective platform out there uh you know in addition to being the best so I think uh yeah I'm definitely seeing the the improve improvements so perfect then any other words before we close the session yeah I just yeah I think uh I I just really appreciate that you know being being on this journey with you

21:27guys I'm I'm really excited about Databricks as as a platform as I've already said and uh you know I think the sky's the limit and for partners like us we um you know again we're we're very very passionate about uh you know continuing on with you guys and uh you know I winning the warehouse and and upgrading Unity catalog and and winning gen use cases I think data rcks is by far the

21:50best platform out there for being able to do all those things on a single unified govern platform so um thank you very much for having me so thank you so much Eddie for being an amazing guest and see you soon all right thank you talk to soon