Databricks MVP: Serverless Compute & Delta Live Tables

0:04hi folks today I'm pleased to have Risha Pand at Databricks MVP here with me as a guest hi rishab Hi youu how are you doing I'm super good super excited after yesterday uh press release that was a massive massive news yeah me too so let's start this

0:29amazing podcast with you me it's great time so I have all this passion to just connecting with these kind of fols so am just trying to connect with you from the last few months as well now I got the opportunity here we are and so tell us more about about you about your role okay so like first of all I would

0:52like to congratulate data breaks for reaching an incredible incredible Milestone like with a valuation of $62 million and this uh this achievement is like a testament to the hard work and an innovation and a vision that a company has consistently shown in data field right also it's amazing to see data RS not only achieve this remarkable valuation not right but also surpass

1:14competitors like there's no flake right solidifying and position as a leader in the industry because the products and solution that datax offers have truly revolutionize how businesses handle big data I mean I also feel proud to work with these kind kind of Technology the new update which we getting right so I mean it's feeling proud with working with datab if I talk about my I mean

1:37background and myself so hi guys I have rishab Pand and I work as a consultant data engineer at salable Technologies a partner of data braks itself right so in my current role uh I leave the data break practices handling all innovations that all Innovation and ensuring we stay at the Forefront of the latest developments in the field right also I'm

2:00also deeply involved in the managing and leading training initiatives within the organization where we deliver high quality targeted trainings to empower our teams to handle and to execute uh good I mean teams to execute projects efficiently and with the best practices that is the most important Point how we can deliver the best the products right

2:22so at SEL I talk about my C as well so at cibel we have a dedicated Learning and Development Department focus on ensuring our Engineers receive top tier training right so this ensures that our projects are delivered on time with the highest standards of optimizations and quality so at some point of time we

2:44almost have 300 400 projects ongoing projects so our training program are designed to scale enabling us to meet the demands of multiple projects at a fast pace without compromising the quality because that is a main point if you're working on a project so how we can deliver the optimizations and a good quality projects right so in terms of data breaks

3:06training I mean you won't believe but we have successfully certified over 500 professionals from our organization and we are looking to add an additional 250 plus Moree professional within the next year so by focusing on this delivery quality trainings and following this standards right we are able to equip our teams

3:28within the knowledge and skills needed to handle complex D projects that that's amazing and I think data Bri Ventures invested in cable Technologies I think yeah last year or maybe 18 months ago uh if I'm not wrong correct correct and I think also if I'm not wrong you're working on some of the

3:50largest uh migrations from all dat warehouses Technologies to yes we have the brick builder Solutions and we have building on top of this more solutions right correct yeah and tell us more about like your first experience with the with data Braes when was it and like what was the

4:13project okay so uh I always had a when I started my career um working with data breaks right so I always had a passion to work on data breaks and uh when it comes to the projects I have worked on I have been involved in so many where I've reviewed the code and do the solutioning but honestly I have lost the track of

4:37the count like how many projects I have been worked on but to give you to give you a few examples I have worked on multiple uh Co complex migration like including moving data from multiple various sources to data brakes right as well as some data varos migrations and a compute migrations right I mean synaps and all that so so a major part of my work is also involved migrating from on

4:59Prime Systems to Cloud environment so in these projects I have designed some unified templates some framework that help reduce code dependency because if you design that UniFi template so the man power will be reduced no that man power will reduce so we can work on the uh two and three people can work on the same so we don't require a full team to

5:21work on us individual project so that is the thing and to streamline the processes that that like the main agenda of design the Frameworks also Implement best practices in terms of code optimization in terms of ECS how we can just give the uh governance to our clients right and how we can just give their security and protocol all the kind of things so these Frameworks have

5:44designed to improve the efficiency maintainability and scalability across the project so it's been a rewarding Journey as each projects offers unique challenges and opportunities to learn to drive in for Innovation and optimiz process in a way that provides a real value to my client so I I'm always Keen to learn new things so that is why I

6:07always choose to new projects and mostly the new updates with the datab are giving right and right now now I also heard of one news that datab has like AWS glue Federation where they can just give the governance to your high meta store tables directly without migrating The Hive to Unity catalog that's the bestest news I heard about in last one

6:29month so um the skin to work in that field as well yeah that's uh so for folks who doesn't know this feature the it was released few days ago where basically you can bring your glue uh catalog and Federated within data brakes and you can grant permission like who can query those Staples of course it's going to be read only but still it's a way to keep

6:55to en to enhance governance or add governance on the top of your glue tables and while maybe waiting for the migration probably to complete or maybe if you want if want to keep glue because you have other systems that that are stick with uh this specific data catalog if you can choose maybe uh one

7:17feature like what would be the feature you're going to choose like from from all the data brakes St stuff uh so I mean to be honest there like not like only one feature I mean we have seen multi multiple features coming in dat from databas so it's hard to choose one because I have some like dld right life tables and right now I'm very much F my

7:40favorite is serverless compute so because in that case the client don't have to worry about the optimizations and all the infra part because that is what databas is giving to you because now that all the optimization that they will keep keep on the things right so you only have to worry about your data migrations and all so serverless

8:02computers very very very Advanced feature and the good feature coming in the market and why DLT I'm curious so DT because previously my first projects when I on the this data break so I got a DT so I'm the first person from organization who have implemented the DT

8:25for the client directly so we have designed the proper framework of DT from taking data from any sources on Prime system to my lake house and from the lake house we have designed the DT framework so we are handling all the kind of things like CDC and we are giving the I mean storing Dimension so before DT if you wanted to achieve the

8:49SCD Logics right so you have to write a code for the same how to you can handle the SD Logics right but in the DT you have that decorators and you have that capability where where you have the applyed changes tables right in that you only have to give the number like one and two what kind of SD you want right so that all the all the things will DT will take care of right so you only have

9:12to worry about how your data looks like how you have to migr the only thing all the remaining things will DT automatically taken care of the debugs the debugging still debugging is the thing that is still uh dat has to work on I mean it's uh still fine because debugging the DT pipeline is some like a kind of hectic process still but it's

9:34fine so you can see the Streamline you can see the logs you can see have the event logs error logs so you can just have all the things but apart from that when you're working on streaming data sets right streaming so you have to give the checkpointing location and all kind of things manually but when you are working on DT you only have to give the part where your M stored so DT will keep take of the checkpointing as well so

9:56it's a find of uh it's a good uh feature this just give me a like a working environment good yeah I got you and for the people who don't know what's DT it's Delta life tables it's the uh managed ETL developed by uh data breaks and you can code in Sequel and uh Python and it has now been

10:18unified and with unity catalog you can only Define uh materialized fs and the streaming tables and what tips would you give to people who just started working on data breaks okay so before going for the tips I would like to share the experience as a database MVP because that will inspire

10:42the other folks out there in the like communities like stack Overflow some are looking for stack Overflow some are looking for gigs for geks so we all we already have a community of databases the same like things right so as a databas MVP it's truly been an incredible achieve m in honor I mean thank to datax so getting the title of datax MVP and being recognized as one of

11:05the top three MVPs from India as well as the youngest across the globe right so in my so I have been proud I've been passionate about using Databricks for my workloads because it has provided me a whole new level of capabilities and enhancement that can have been invaluable to my growth also going the forward in like upcoming years so beyond

11:27my work I've also made it to a point to actively contribute to the community of databas right I have solved more than 500 problems and have been fortunate to be recognized as a databas community champion as well so it's a rewarding experience to help others while also learning from the same uh uh community so this journey has fully and truly F

11:49been fulfilling and the MVP title just add to that excitement that you can feel proud to work on this because if you're working on any kind of things so if you been awarded so you get more motiv motivation to work on the same right so this is what I would want to share my experience and if I could talk about like what advice I could give the uh defox who are working and who are using data brakes and who are using some other

12:12tools as well I'm not stopping you guys but yeah so I just wanted to share that what advice I want to give so my advice to the peers would be stay curious the first important point is to stay curious and always be willing to learn the tech landscape is evolving every day rapidly especially in areas like data engineering Cloud Technologies and

12:35platform like data brakes so EMP strange be open to new tools and Frameworks and can design your try to right and never the power of community the datax community has been a great source for me and actively participating in it has not only helped me grow but also allowed me to help others also focus on

12:57understanding the code principles behind the Technologies you work with rather than just tools themselves don't focus on the theoretical concept because that's a very basic one but if you don't have I mean always try to work on the things as a handson because when you get the Hands-On part then only you will be getting to understand what exactly that feature mean to you right so yeah that's

13:20what and I think people may be curious like how did you end up joining the MVP program like did you apply or someone reached out to you okay that's interesting question so uh I was just contributing the community from last three or four years right in the community and have been solved so many

13:42problems but I think that to be uh recognized as a MVP you would you should have some good connections who can nominate you guys and for me my Company CEO Mr andr kala has nominated me for this program and then I have showed what skills and techs I have and then yeah

14:05the remaining result is in your friend that that's amazing and you are brilliant I've been like following some of your post with other MVPs Julia hos like yeah super super long list it's just uh amazing and also like um

14:29I'm curious to know like have you like to develop your skills have you relies on have you relied on data breakes Community or maybe like like datab breakes Community or data breakes Academy like how do you how how uh was you able or were you able I mean sorry how were you able to improve your data break skills because I've seen like so

14:51many people like asking those questions how can I should I be certified should I take those courses I don't know should I buy books what's okay so what would be your advice with this answer I'm going to just uh reduce the selling courses out there in the LinkedIn right because I previous uh preferbly I don't believe on this kind of courses coming in the market because

15:14when you already have a databas Partners platform and like and a customer as well so whatever the courses we have in the data itself those courses are tremendous of level right because when I have I have no understanding of data break so I only focus on the coures that databas provides the Hands-On Labs the instructor trainings from Van and I mean

15:37the wanket was the one of the favorite trainer I have been trained with right so we have multiple trainer as well from datab so if you wanted to learn something you can go for the courses that is like Udi courses and we have some members who are just trading course on you me right but before that I would recommend you guys that to focus on the database platform as well because we are

16:00getting a good courses there we are getting the updated courses every quarter I mean right so we have some new features so we have been involved that also in the Advanced Data engineering with data so yeah that's it so this keeps me learning and if I talk about my community uh community so from the community when you let's suppose if you don't know something

16:22about data briak so and also if you join as a new guy and if you wanted to work with data break so if you are going for for the community of rars you will be seeing some question out there like what exactly the uh candidates are facing some problems in the projects while projects so they posted their questions in the community and then I'm wondering that I many solutions are coming in that

16:46post right so when many solutions are coming in that post so you will have a chance to work on all those multiple Solutions you can Implement those Solutions and then you can get idea and you can more get the idea which solution will the optimize one so this is how I have learned in this three or four years because if I go for the course the theoretical knowledge I will get right

17:09but until and unless if you're not ready for the Hands-On part then it's of no use because anyhow you have to work for the projects any how you have to design the Frameworks I mean this is for Architects as well this is for solution architect the data engineers and all of the uh fox out there if you wanted to really wanted to work for the things you have to put the handsome and that is coming in like Direct in the

17:31datab well that was well well summarized and do you have any final words before we close the this session okay I so this is the session in which I have shared my experience so my question to you that because I got the opportunity to talk to you so I would like to you that how is your experience

17:53work ATA briak how you have evolved from nothing to this much of capabilities you are now having as a te person I mean you are never bored that's for sure because you always have something new to learn like you can never say yes I mastered the platform now that's wrong if you if you're thinking that you mastering the platform

18:16you are completely missing something because the platform keeps evolving and correct I guess I joined the company maybe three and a half years ago and uh I've seen a lot of features like Unity catalog Delta life tables data brick SQL uh Lake View aibi Lake flow um the

18:38governance part system tables data RS asset bundles workflow online tables like just insane it's just insane and but I see that now you can build a complete data platform from to end just

19:00by using data braas of course you can use Partners if you are leveraging Partners or other Technologies but you can really develop something end to end just using data braks like I remember one of my colleagues built u a data Brak like data deploy like streamate application on data brakes and then they used uh just a camera to take pictures

19:23when once you take this picture this picture is uploaded to data volume you handle the governance there then you're using an llm to generate some like super power like give you some strengths like I know be similar to Hulk or I don't know Spider-Man or any other person and then this picture displayed and when you see like the components here it's just Databricks like volumes Unity catalog

19:48uh autoloader uh LMS and of course data braks apps so everything is hosted and of course sever comput as well which means everything now like can be developed on the platform and the platform is getting better and better like we're adding more features like Lake flow to injust from some key uh SAS platform or databases

20:13with like lot of things make things easier and really the goal I think and that was what was the goal of the company democratize uh access to data and Ai and I still believe that the most important thing is not to think about the technology but the technology can help you to reach the goal because you can always use fancy features but what's the goal behind it the goal is just to

20:36simplify or to reach either democratize access the data increase the ER the return of investment reduce the cost increase benefits so always start with a goal and not use the feature just because it's a fancy one I think this is something this is something important use it when it's bringing a value to

20:59you and keep and the other thing will be keep learning because technology keeps evolving like what what we had last year it's no longer valid now we have a lot of things that are completely different like you see regarding the llm like fine tuning now have things that can generate questions for you evaluations Benchmark like things are going fast so just try

21:22to keep up with all the techn all the things that are evolving like day after day and it's not only but for data it applies for all the other vendors and partners and Technologies in open source and propriatary one it's great to learn about your experience so yeah last but not the least you forget about AI JY as well so

21:47that's also a good one in data breaks yeah you just bring a table ask a question and you you have like immediately some answers there for you of course you can make some mistakes but again it's up to you to tune it and add some metadata to tweak it and then of course after three three or four trials like test you can pretty sure that he

22:10can handle maybe more than what you can expect so just give it a try correct correct so guys uh so guys if you haven't connected with us on L yet right so this uh we are working on D platform so yeah we have a use as well so if you haven't connected we can you can get on let me as well perfect thank you so much uh rishab

22:36and see you soon same here thank you