Unity Catalog Migration: Lessons from Databricks MVP Ajay

0:05hey we are back with AJ it's part of the MVP series we started uh recently and hi AJ I have uh I'm super happy to have you can you tell us more about you yep thanks thanks you so I'm very happy to join this call myself a Pand I am uh

0:30associate manager at Tom having 6 years of experience also I'm the community champion as well as the Databricks MVP so yep and what's your background are you a data engineer data scientist yep so I come with the uh data engineering background so uh mostly work

0:52with the data B only so had a lot of uh done lot of the project in a uh databas in a engineering domain currently I'm working as you can say solution architect in a data B amazing and tell me like what was your feeling on how how did you know uh how did you knew that you are was selected

1:14to be part of the MVP program right so uh basically I used uh mostly databas Community where uh I have uh contributed a lot of the things so basically the community one from where I got to know that I've been selected for the datab MVP so in a datab community uh you will get a good options to

1:37contribute as well as help others and you can also take a help from the database community so there lot of the data Engineers peoples as well as the databas peoples are there so you can if you have any questions or you're facing any issue that you're not able to solve it you can just post in a database community and then the people will be there and they will be helping a lot with the thing so I have just contributed I'm the member of the

1:59databas uh Community from last 2 years from the beginning itself you can say from there I have contributed lot of the you can say the uh answer lot of the questions I have helped the multiple people to solve their you can say the hug cases so from there itself I have been got invite from uh the MVP part it was ear this Bon one so now it's just renamed to datab MVP so I got a from

2:22invite from there itself I think if I'm not wrong uh I've seen your name because suita or KES must have uh told like shared one of your one of your articles or uh your post on on data bre Community but that's good thing this is this is good that you mentioned the data bre Community because few

2:44people I mean it's not that famous but it's really the place to go where you have if you have any question you have a lot of expert there they're going to answer the questions can have documentation a lot of things and keep in mind that maybe up to 60% of the questions you had on the top of your mind has already be asked H posted and have an answer so rather using than

3:07using any llm just go there you'll have your answer yes yes that's and AJ when when was your first time you used uh like uh data braas when did you use data breas for the first time so I started my career in a datax only so when I joined in you can say the 2020 so uh

3:30uh I was first one where I use databig you can say the community account where we'll get yeah Community right so from there itself I started my journey then uh I just uh earlier uh I got a training from the datab B itself so it was starting so uh that time we were getting the free trainings from the databas like the uh so from there I itself I started

3:54then I explored the multiple thing with the help of the databas and then multiple project in a databas then I got a better idea about how it's working and what are the things then I just started comparing with the other you can say the uh competitor which is the Snowflake and then databas now like then databas doing very good compared to the other you can say the competitor so that's why I chose

4:18datab I use it uh I think three years before you for the first time I was in 20 2017 I Was preparing my master degree and we got a project on spark and that's where I I found the data briak Community it was free Compu I was like oh I don't need to have a VM I don't need to install Hado

4:40spark this is revolutionary and back in that time we had only notebooks and Spark that's it yes I think even the orchestration didn't exist back back in that time I think so yeah that was amazing so since you knew data breaks uh since 2020 I mean are you not shocked by by by the

5:04pace of innovation uh data break is bringing those numerous features every quarter what's what's sometimes you can like you're wondering like if you you compare the platform like four years ago what would be your what would be your uh your biggest outtake let's say uh I would say

5:27it's totally new now so in every week you will say there is a new features right and that's uh helping a lot with the people and data engineer so when I started using the data Bas it was generally the normal computer that we were using the orchestration was not much mature that time the workflow one then uh it's getting mature day by day

5:49now it's y jobs yes yes so now it's very mature now we can rely on that one ear we were using this other the other tool to orchestrate the databas job so we were mostly using the ad of AO data Factory on AO to orchestrate our job now we have discarded this one now databas is providing much better way to handle the

6:11job as As schedule the job as well as uh now we have a serverless uh compute where we uh don't have to worry much about the you can say the infra part databas is managing and it's very good now and we are using also that one also we have uh seen multiple things like now we have a gen when the J boosted now datab is the first one to implement the

6:33things now also we are using the Gen and implementing and we are suggesting customers to use that one so it is very good as well as now uh we are getting lot of the things as you can see this L flow connect now are there so it is right UI based and it's very simple to use so now if anyone don't know the coding and other stuff they can directly click on the connect and direct they can

6:55use all the things so earlier uh the uh you can say the there was no much on a databas you can say dashboard part now we have a dashboard and the reporting tool now also in a databas only so we can directly Implement uh reporting as well on this thing so yeah it's change a lot but it's very mature now yeah and also as you mentioned like

7:18we are really simplifying the workloads you talked about Lake flow so you don't need to have a very let's say Advanced knowledge on data engineering to set up your pipeline you just select the platform whether it's SAS or SAS platform or SQL database you enter the credentials and then the

7:40data is here in datab braase and then if you want to do some transformation it's declarative transformation part of DLT so the CDC is very easy to use and then you can just serve this data to as you mentioned aibi for example dashboards you want to build your dashboards it's very easy I am very dumb when comes to building dashboards but with AI I just add like use the llm

8:04integration to ask like hey build this dashboard here yep and that's it you know it's very very easy uh very easy to use and like the goal I think was always to democratize access to data and AI to everyone right and uh mostly I forgot to mention this the UN cat log that was not

8:27there course right so is introduced like two or 3 years back and like it CH the way we are that we are using the databas it's totally now changed so now we have a full control over the datab BX working space as well as the you can say the data as well so now we can see all the lineage all the access at once if you want to do give the some access to the users directly then we can go and select

8:49that things if you want to track any data lineage suppose we have a dashboard now want to take how what are the particular base table that we have used what are the columns that that we have used so we can directly T back to the you can say the source of that things so yes the UN catalog is the one that changes a lot yeah indeed and to be honest when I

9:11joined the company Unity catalog was not there and I'm still wondering like how did we survive before without Unity catalog like I think we used to have only akles and akles was something you you can set on the Clusters but what if you create another cluster you bypass this security it's

9:33like I mean how did we do like and now that's I don't think any company can can afford to avoid missing or not using un catalog because the governance and data data security of the data is very very critical uh and important and I um I think you must have worked on several uh Unity migrations I believe right right I

9:57have worked lot of the my as well from hype to the UN log so yes it's help users and users also very uh happy to see their now they have much control over the data as well as the resources that they have and they can do lot other things with the UN cat log now we have a system tables where we can track all the things as well as the cost of the job level as well so yes it's helping a lot

10:22yeah it's part of the phop parts where we can deploy a dashboard to track the consumption we have the workflows one includes several and you can try it you can use it to optimize your workloads it's super super amazing but since you mentioned the migration AJ what are the best practices or the recommendation you will give to anyone who is currently now working on a Unity

10:44catalog migration based on their experience yes so uh the you can say the main part is like when we are migrating from the hype to the uh you can say the unate log then we have to be very sure that uh we don't miss anything so lot of the jobs will be there lot of the schedule will be there then we have we have to you can say the

11:06first make all the things that are there in the uh client and that we are not missing it so sometimes if happen then we are doing some migration we miss the things and at the end of the things then we get to know that this is missed so that will be the most uh you can say the most things that you have to first list down all the things that are there and you have to be very sure that all the things you are migrating is up to as

11:29well as the lineage you could say the history of the data all the things should be migrated from the hype to the uh you can say the UN cat log so those things are very necessary when we are migrating from hype to the UN cat log yeah and since mentioned about like listing everything we have we I think we have we released uh an Excel sheet where

11:53we have all the steps and all the things we need to follow to make sure uh to have a specific like successful catalog migration and you have also tools like ucx ucx yeah to migrate your assets in a very simple and easy way and the latest version is just just massive uh and tell me J have you tested

12:15you the you the UC open source version or not yet uh yes I just got a chance to access that one so I just uh I'm just doing the PC in my you can say the system so just connecting with our existing database to see how uh it's working so we have not worked a lot of things but like we are just getting started with this uh the Open Source One

12:38yeah so the V so I really advise you to test the v02 uh like a lot of things like integration with spark uh with ML flow a lot of like UI a lot of a lot of exciting things uh uh are over there and I also another question for you like let's suppose I am a new day engineer and I will learn about data brace what

13:02are the steps I need to follow yes so first starting with the Databricks will be very good uh if you have the uh you can directly access the customer Academy that we have in a Databricks so it is totally free to start with data braks you will get a lot of things lot of the courses are there which is totally free for us also uh now

13:24uh datax uh has released this uh you can say the uh trial version of the datab where you will get a 400 you can say you can you can say dollars to just start with the datab Bas where you will get all flexibility like in a community addition you won't be able to create up you can say the workflow there's a much rest restriction on that one but when you now starting with the databas it is

13:46very easy you you don't have to worry about the any Cloud platform directly you can sign up with the serverless and you will get a $400 and now you can start and you will have a full flexibility on doing anything on that particular work space so now yes it is very good to start with the datab B with the serverless work space um I'm just speechless I mean you

14:10pitch you pitch it super super super super well and and one other question like uh this like with regards of the data breakes Community have you ever uh use data brakes uh Academy yes I have used data breakes Academy so so uh earlier uh we get a chance to work on this uh datax Academy

14:34it's help a lot so uh uh currently uh I have taken a lot of the courses from there so which is basically for the llm and gen so it's it's help a lot to uh do the PC as well as go to the production so yes I have used that one amazing and since you mentioned the uh the Gen part so

14:59but before before before we move to the Gen tell us more about let's say one of your favorite project you worked on for the last couple maybe two years yes so uh the project which is I'm working currently is will be the favorite one so here what what what we are doing we are uh we have onr system uh so we are we are doing all the

15:22necessary things but now what we are doing we are moving to the uh datab Bas from starting itself so lot of the things earlier we have planned to do uh the orchestra with the ADF to take a data from the SQL server and then do the transformation in the databas so now uh what we have done we have just do the PC and other step and then we have

15:44discarded the ad because we don't think the ADF is now uh much uh helping because we have a lot of thing in a databas only so we are fully uh you can say the deployed on a databas and now here we are using all the latest thing that database releasing so we have gener implemented jna we are also implementing L flow we have a you can say the we are

16:06using the llm as well we are using the machine learning so all of the things are there and now we are fully you can say the uh utilizing all the things that are there with the part of the datab Bas one yeah that that makes sense that makes sense and by the way since you mentioned like the data bra Academy and gen training so just for those who are

16:28watching uh this interview uh the training on the Gen is not focused on datab break Technologies but mainly on open source one so it's a very good way to start learning uh about the Gen but we have also another I think Advanced course on the llms like what's the vector search how does it work how can we do tokenization and all the other

16:52things that you you should know before getting started and uh with the llm uh with this llm or your with your lrm journey exactly so that course helps a lot uh with the open source as well as it's give a reference to the databas uh you can say the services so now uh databas providing all the services that are required to implement the uh jna so

17:16it could be anything uh so you don't have too much uh write the code now so it's uh datab abstracted all the things and providing easy to use the things so if you know the little things you can take that particular course it will help you all the things you can all the resources that we are going to use all the tech that we are going to use so it's help a lot to implement the things and

17:39yep yeah you're you're totally right and I have uh one final question for you before we close what's your favorite data breaks features feature H so I know it's tough question you only can choose one option so of course all right so currently it's

18:05l flow connect that that I can say uh here uh we'll get what we are getting we are getting the all the things are implemented inherited you can say so we are just getting the UI just connecting selecting the source and all done CDC all the things incremental load all as managed by data braks so it is very easy

18:27to use for our you can say the non technical developers or other part and it's helping a lot to not worry about the you can say the change in the data because data will handle that things internally and here uh we are using the DT underneath it's using the reality and using the server list so you don't we don't have to worry about the data

18:49optimization uh infra part so all the things are abstracted now and it's very very easy to use yeah you're right and I think I I guess you chose the I think it's one of the most awaited features uh this year and bil who is a senior director behind this project must be very very happy but yeah thank you

19:13thank you so much AJ for all uh your insights and I will for folks who are watching this recording I will add the link AJ LinkedIn in order if you want to reach out to him or follow him he's sharing a lot of interesting articles so this he is the person to follow right right right thank you thank

19:36you sir thank youj