Databricks MVP: Data Engineering, GenAI & Governance Best Practices

0:04Hey, welcome back to this M meet the MVP. Today I have Rajnish. Hi Raj. Hey Ysef, how are you today? Great to see you, man. Likewise. Likewise. I'm pleased to have you uh with me today. I think you're going to be the u the sixth per the sixth MVP I interview uh for uh

0:26my YouTube channel. So can you please introduce yourself to the community? Sure. Hey guys, my name is Rajnish Koshik. I am based out of Toronto. Uh I have 25 years of industry experience. I'm a datab bricks MVP, Microsoft MVP, datab brick champion. Uh I have uh I'm currently working in a company called Virtuosa Corporation as a

0:49data and AI uh head for datab bricks practice. Um I love coding. Uh I'm a big fan of AI and GI especially. Um apart from that uh I love hiking uh sing uh you know listening old songs. These are my hobbies. Uh great to connect uh with all of you

1:12and you mentioned that you are you are collecting a lot of like MVP sit. I know for the Microsoft MVP you need to take some certifications and then at least I think two and then you need to apply through a website to be um like MVP and also you need to be active in the community. How did you end up being an

1:34MVP at Databricks? Uh how did your journey started? Yeah, this is a very interesting journey you know but I uh I have been you know very actively engaging in the communities for such a long time right and uh I love helping people engaging in technical forums uh and and you know that really helped me as well to grow in the career because

1:57when you help others you tend to learn many things right and uh when I was going through this journey u you know suddenly one fun day I got a invitation for data brick MVP uh because uh looks like people started noticing my uh community work uh especially couple of blogs u which were really highlighted in the datab bricks uh as well like

2:21snowflake versus uh datab brick uh comparison blog and similar blogs like data databick a bi related specific blogs and then that is where uh my journey to uh become MVP for databick started and I'm really glad because this is a great opportunity for MV MVPs to u you know connect with similar kind of people uh similar interest learn from

2:43each other and uh contribute meaningfully to this great community because you know when you contribute there you grow as an individual you grow professionally you learn from each other right so this is a great community I love it I love datab brick as a product I love this community and u this is a great discussion we are having you know every couple of days there are new features rolling out from datab bricks

3:06and we are learning how data brick is evolving as a platform from where it was now where it is today. It's amazing and you mentioned that you are very active in several uh communities. So were you engaging in maybe datab bricks community, Reddit, LinkedIn like um which platform are you currently using to engage with your community?

3:28Absolutely. So I'm actually using LinkedIn, I'm using YouTube, I have my own blog uh rejik.com. I have my own YouTube channel um you know on the same name and then uh I also participate in many events hosted by datab bricks. uh also try to you know you know try all the latest and

3:50greatest features u released by Databricks as and when it is released and then uh try to implement those uh features in my you know environment where I'm working right to exploit you know how can I use this feature in my current uh environment current project right so that is where my whole interest is always

4:12can you tell us more about your first uh data brick experience when was it and maybe you can tell us more about the use case you were working on. Yep. So I fell in love with Databricks almost I would say that five to 6 years ago. You know we were working for a large banking customer in North America. We were migrating a very critical reporting

4:35platform um from uh PowerBI report server to the PowerBI cloud and at some point of time uh because of the very u biggest you know the SLA problems right the performance issues which we were uh coming across with these reports at some point of time we decided that we could

4:57have used datab bricks because datab brick is a sparkbased platform it can scale up uh to the need of this kind of issues right it can help us to tune the performance and that is where I started exploring what is datab bricks let me explore it um that time databick was not as mature as it is today but then uh we I mean I realized that this

5:20platform has lots of good potential you know you know it can scale to any level it can um it can also provide this kind of performance which we are looking for right and easily it can attain that uh you know SLA is very very easily it can do that and that is where I started exploring it uh started learning it trying it multiple places in the PC's

5:44and that is where my I fell in love with this platform I love your your your story and um what about your specialty like are you more focusing on data engineering part data warehousing or maybe the ML part I love GI part um a lot. I I'm a big fan of Genai. Um my my journey started by the

6:06way uh with data engineering. Uh then I I like you know I saw that uh how easy it is to do the data engineering in Databricks right um it's very very easy to do the customizable pipelines. Uh there is no unnecessary uh you know uh interruption in terms of many different

6:28uh platform what they provide today. Right. So it's pretty straightforward way to do the Python coding and pispar coding and then get it achieved right whatever you want to achieve it's very easy to do that then I u like recently last one one and a half year I started using ji a lot right creating agentic AI um multi- aent lang chain that is one of

6:51my favorite semantic kernel is another one uh geni evolution is another aspect to it um uh yeah I mean this this is really great especially these features are amazing. Yeah, I think you will have plenty of things to to discover with the uh with the latest feature the a like agent framework ML evaluation extern like like

7:14AI gateway to have models save AI guard rails a bunch of as a judge evolution as a judge. Yeah. Yeah. Exactly. And you know I love AutoML man. This is really great. AutoML amazing democratizing the whole you know AI ML amazingly very like is one of the niche

7:37in the uh machine learning. So I love that. Yeah. And especially with the uh with the glassbox approach where you select your model. Yeah. I mean you select the data then whether like classification regression yeah forecasting select the metrics and let it train and then gives you like all the notebooks that were choose to train with how the sampling did happen the

7:59parameters of the model and how to deploy this model and you can like can be a starting point for like rather than doing things from scratch absolutely just give you like books and then start editing and also providing you with the EDA the like the basic EDA and of course if you want to deep dive you can do it on your own and it is done in such a way that if you are not a u very like

8:22you know you you don't know much about uh ML models and all that still you can still use this feature right and especially autoML is something where you can use it very easily even if you are not deep down into this kind of technology you don't know much about it still you can use it for the fundamental classification regression right casting things so that is one thing which I really like because Most you'll find

8:46many people who don't know machine learning in depth but they want to use machine learning that is where they can start their journey. Yeah. And just to go back to your role like as a head of uh the Databricks uh practice. So technically what's your role? Is it like maybe to win like uh PC's or maybe to draft the architectures? Yep. Can you

9:09tell us more about your role? Yeah. So this is interesting question. So what we do we virtuous is an engineering first organization right we have a very huge um like u customer landscape where we try to implement the data brick u as a platform adopt the data brick as a platform um we have recently also developed our own accelerator for unity

9:31catalog migration right which is u uh pretty good in terms of you know migrating end to end from metal store to unity catalog we also uh implement the data engineering ing capabilities for our customer machine learning models specialized machine learning capabilities which we try to uh deliver and develop. My role is to you know uh 50% of my uh work uh is focused on the

9:56uh you know proposals RFPs right and then then 50% also I try to be very hands-on I try to do my coding try to see what are the brand new features upcoming and then how can we utilize that so these are the two aspect of what I do I have a a team who reports into me uh where we you know uh try to develop the new set of accelerators

10:19Like one example is our builder solution which we recently developed for Unity catalog migration. Yeah, very very interesting and uh to go back to the uh the platform can you tell us more maybe uh your top three features? So uh top three feature from databick is um one single platform for doing the

10:42geni uh doing the dashboarding reporting right and uh and you know the good part about the datab brick is that you don't need a separate licenses like for example other platforms you need a separate license for each of these features right in datab brick everything is integrated there and very well very nicely integrated number one which I really like it right And second feature

11:06which I love about data brick is the performance. So far I have not seen this kind of pispark performance in most of the platform right uh I love it because datab brick it can scale very easily but not just that the pi spark implementation which datab brick has done it's amazing you know nobody can beat it I can I can tell you that I mean I have seen it um projects running in

11:29different different platforms but when it comes to the performance uh which datab provides it's it's amazing right so these are two Second feature. Um third feature is uh and and actually it's not about data. The way the uh database is now rolling out the brand new features very quickly. For example, I'm a huge fan of the AIBI and clean

11:52room. Recently they I have actually uh published a blog as well as a video on clean room and AIBI right. So they are very relevant features for today's world. Right? Unity catalog is a gamecher. I love it. And uh you know I always think that why Unity catalog came so late right governance is a lead of the

12:15hour and everywhere we try to use but good part about data brick is that unity catalog is single onetop solution inbuilt into data brick you don't have to purchase it outside so it's integrated real integrated environment and u another aspect to data brick which I really uh appreciate is the open um like it's it's a open-source technology

12:39right you don't have to do the vendor lockin for all these features like for example last year I was attending the data summit they have made this unity catalog as open source in front of me it was so good to see that kind of event right amazing you know so this the company is going in the right direction and uh the way they are launching the

13:02new features partnerships I really love it and maybe if you have some uh best practices that you can share with uh the viewers for example if they want to start it maybe project in Databricks what they what should they consider like any best person can be good absolutely so first uh practice is that always try

13:25to do a reusable frameworks try to develop your own framework which can be reused in your application right and then like for example for injection perspective If you are a data engineer, you might be using some uh you know you are going to create some pipelines, right? So don't just directly jump to create a pipeline in pispark directly. Create a framework around it and you

13:48know make a framework which is very customizable, reusable, right? For different scenarios. For example, if you create a metadata training framework where you can add all that injection capability in the metadata and then run it, that makes it very easy for you to write uh the code very faster way, right? Uh that is one thing which I

14:11always prefer, right? Um and second thing is that u always u make sure that governance is your first approach always because if you don't think about governance right and then when you come back after some time you will feel that you have lost so many things in that you you have to be very careful about uh how you set up the environment how you um do

14:34the end to end governance like whether it is a IML or any other like pipelines right lineage all those things you need to plan before you start the development not afterward right because generally as a developers people have a tendency to directly jump to the code uh and then start figuring it out after we are done with the development right but I think

14:56governance is a very important and critical aspect to it and u I'm also part of a initiative for CIS benchmarking for datab bricks where I'm working very closely with um you know CIS team to create a security benchmark for datab bricks So always you know keep security as your first priority right because all your customer data uh is very very sensit

15:20could be very sensitive and then you need to be very careful when you you know expose it. So use the best practices security best practices. Uh that is these two things very very critical. Yeah, that makes sense and this is this is very very important and as you mentioned don't get started immediately but try to think about design and of course uh then start like learn like

15:45implementing and always make sure to think about the security because it's super super important. It's security comes first. Yes. Not the opposite not developed and then think about security because this may be a wrong uh wrong approach. Yep. Yep. And according to you, what's the best way

16:07uh to get started with Databricks? Wow. Like what what would you recommend? I know it's a tough question, but what do you recommend? Like is it I know like people will say learn by doing that's for sure but it's not always possible for for some folks. So what would you advise them? I think uh my approach have been that try you know so don't just theoretically

16:30learn these things um learn a very small amount of the theory and then apply it more practically you know learn small snippets some feature and then try to apply it in the databic open your Azure or AWS console create a databick instance try to run some code in this right because that's real learning right theory certification is okay. It will

16:54give you some foundation but real uh experience comes from the you know by working on those features implementing those features right and uh good part in today's environment is you have a lots of great tools you know you have a chat GPT you have other tools available at your disposal right when I started my career 25 years ago the those tools were

17:17not available right so uh you know now we have so many tools you have you Even even datab brick provides a AI assistant very easy to um you know learn uh very very easy to ask the questions and clarify it there and then. So I would suggest that uh put a learning plan right start with a fundamentals first make your

17:40fundamentals strong and then go to the next step where you know keep incrementally learning those aspects and then always try to get certified because that will add value to you as a at a professional front. Because when people see um you are a certified u databic certified uh professional then definitely it gives a different kind of confidence to the

18:02customers right if you are especially you are going to apply your skills to the job. So these are the three aspects I always try to you know uh focus on as you know an individual or a professional and before we close can you tell us more because you mentioned that you attended the data and AI summit last year by the way I I was there too and can you tell

18:26the viewers like like if they are still struggling whether they need to go or they need to attend or not if you can like tell them maybe your experience like and How how like describe your experience because I know it's super super hard to do this kind of exercise but I want to have like your opinion. I mean this is one of the event you should never miss. Let me tell you that very

18:49clearly. It is not to be missed. This is a great event. Why it is a great event? Because under one amola in one place you will find all the AI and ji or machine learning enthusiast in the same place. great sessions um very unique sessions right very unique uh wonderful sessions you will find uh very thoughtprovoking interesting sessions and then not just

19:11sessions you will get a chance to interact with those people right you will see what is coming in the AI space what is new how people are approaching AI uh data engineering what are the different new dimensions coming up all those aspects and that can only happen once you interact uh with uh with the folks there right that is why this event is a must

19:34attend event and last year experience was amazing I saw unity catalog uh getting open sourced in front of me it's a historical event I I I was witnessing that event in front of me and I still have goosebump when I saw that uh when I remember that event it was really really great you know that kind of people who come there right that the kind of people

19:56who attend that event present their sessions Berkeley you know AI professors they they share their uh learnings with the uh you know with the in the conference that kind of learning is and this is a new upcoming technology new inventions uh new horizons in the technology which is coming up and and you are witnessing it

20:19right we want to miss that so I love that I love that thank you so much that was clear I think now you know like if you're able to attend then go for it. If not, you can still register for to watch the online webinars and it's free of course. Uh thank you so much Rajn. It was super uh

20:41interesting talk. I was really uh glad to uh to have this opportunity to talk with you and why not maybe we're going to meet at DICE if you if you intend to plan to go this year again. Absolutely. Absolutely. Looking forward for it. Thank you so much Yousef for your time. really appreciate that and thanks for hosting me. I love your webcast. I love the your channel. Um you bring new MVPs.

21:05It's a great opportunity to connect with fellow MVPs and learn from each other their experiences. So it's a great event. I love that and I love your channel. Thank you so much. Thank you. Bye. Yeah. Bye.