Databricks Data Engineer: Career Path & Best Practices

0:05hey hey welcome back in this MVP Series today I have the pleasure to host Daniel Sal hi Daniel hey hello I'm really excited to have you here but first can you tell us more about you uh first of all thank you for having me here I'm Daniel I work as AI

0:30data engineer I'm like almost a decade in a business delivering the uh data engineering solutions for the clients so my background is like not fully data engineering I as I was also doing some software engineering before like a couple of years ago but yeah I've

0:52started mostly with a data engineering like my first professional experience was working with EXO that's that was kind of cool stuff doing the reports in Excel but it was exhausting in the end and after that I've came through working with on Prem database Solutions so some Cloud Solutions on Google Cloud platform with

1:16big query airflow and yeah for the last three years I've been working with a Databricks and yeah delivering D house to the clients yeah I think since you mentioned Excel josway will be very happy to have a talk with you yeah like Excel to the to the limit

1:39so I know a lot about Exel and also I'm really impressed by like the experience you do have like working with so many uh Technologies so many so many tools and I think I think the first one like can you tell us like how did you end up uh working or like uh

2:01developing on data breaks in one of my previous company like we're using mostly Azure stock and then after some time there was just a need to start doing project projects in a data breaks we found a client we had a client and yeah we had to start with a data breaks and since I

2:23do like to write a code it was kind of fun to me and I like I like this product and I really enjoyed working with that and since you mentioned that you started working data braas like uh three years three years ago I'm not wrong so I guess you notice the uh the amount of

2:45change like the introduction of unity catalog uh Dela life tables fature store you've seen like how the platform uh like releas so many many many features yeah it evolves so fast that sometimes it's really too hard to track what change what's new but yeah I think

3:07that the most important feature that was introduced is the is the unity catalog because thanks to the unity catalog you can keep everything in one place you don't have to set up like other tools and stuff you can have everything in one place and that's kind of cool and I think that this is the way Databricks is developing itself to have everything

3:29in one place with a simplified architecture so you don't need to take other bricks and build like a a whole block of the uh of the tools apps and infrastructure yeah you're you're completely right and and you know what I um I remember like when we did no before

3:53before that like when I join data braks I think since I joined the company I've never felt bored because every day literally like every week we have new release and there is like a joke we're always saying like whenever we go on like holidays once you're back like the platform always there is always something change like Chang in the platform like it's can be class maybe

4:16new release maybe re rebranding a product or maybe adding new features so it's always like you know when you're back need to keep up with all the changes and I don't think you may have this opportunity Maybe if you're like I didn't have this chance in my previous in my two previous uh rules um yeah so that's why I start

4:38Monday with a cup of coffee and checking the release notes and that's kind of cool with a with a previous technologies that I've used I did not care about change lock I do not care what's what's going to change what's happening and the uh the amount of releases the releases itself were not so frequent and with the data breaks it's like every every week

5:00you've got something new or you can enroll to the previous and test something that's not yet publicly available and that's kind of cool I think that this is because Databricks is listening to the community and they are adapting to the clients and and people needs yeah yeah I I I would totally

5:24agree and like if you I don't know if you're aware but the number of private previews that are like currently rolling out it's just insane insane like the number of features like it's sometimes also super hard for us like to keep track of all those changes like you need to have like cheat with all those all those and also I have um like couple

5:48questions for you I I've been people I've seen people asking same question like what's the easiest way to learn or to master maybe data breaks I know it's like it's a large topic because I have a lot of like breaks can be engineering analyst ml but since your data engineer what would be your advice for aspiring

6:11maybe data Engineers who want to develop or increase their knowledge on the platform that's actually a good question because for the last three years I've been also technically interviewing people in my in the companies that I work for so what I find common in in the candidates is that they don't know or don't fully

6:33understand the basics so they say that they know Databricks but they don't know about the spark Basics so I would start learning from the basic how spark Works what are the backgrounds how to check performance Etc then moving on to the uh Delta format as it's the main format for the

6:55Databricks and then going Higher and Higher Learning Unity catalog how it works what kind of issues we might have in there and yeah definitely learning the basics not jumping from one technology to another because I also saw so a lot of folks like they knew a lot of Technologies but

7:20it was like they knew a little bit about one and another but they were not able to master any of them and that is a that is that is a big issue because if you master one technology then moving to another all you need to do is to learn the differences you've got the solid fundamentals to do that so yeah focus on

7:44on one technology learn how it's working and then move on to another if you if you've got a need yeah and and I I think that this problem is coming from the fact like uh whenever I check the descriptions in the rules that I that I see in LinkedIn they always mention of a huge number of like

8:06stack you can find like data braks uh airflow kfka sometimes you're going to find the snowflake or click pobi like it's just insane and that's I think that's what motivates people like say okay maybe I need to try to learn like small piece for every single technology and I think you well as you

8:28mentioned it's completely wrong you should just try to focus on one technology start with the basics as you mentioned like start with the Apachi spark Delta Lake then move to Databricks if you good then you can move I don't know to other tools like click or maybe airflow but just don't tackle everything at the same time think it's just going to be lost yeah the truth is

8:51you won't be able to know everything and also you won't be able to answer all of the questions during the uh technical interview that's that's simply it's not possible but I think that also the issue is with the uh big Consulting companies that all they care is the head count and the number of projects so they are like

9:15hiring unexperienced Engineers they are throwing them into the project they have to do their work without understanding what's going on otherwise they will be fired that's that's how truth and yeah that's that's why why it's hard for them to learn the basics because they don't know where to

9:40start yes sorry my apologies start here yes I I I I totally like go go through point and if you can tell us a bit more maybe about maybe one of the or two of the most exciting projects you did work on like recently if you you can tell us more

10:02about maybe the architecture how what was the experience what was the objective every project is uh kind of interesting and challenging in some ways even though some of them might be similar there are main differences mostly in a people that you work with but interesting projects maybe one

10:25of them was uh we were creating uh self surface data platform in a Databricks it was it was done like this we have a team of the of the uh of the engineers but kind of more business Engineers that they wanted to onboard a new data source they didn't know much about data

10:48engineering they didn't know spark python or they knew the basics so we've prepared them a framework where all they had to do is to fill in the uh fill in the configuration and the spark was doing like everything for them the data processing creating tables creating

11:10schemas data processing moving the data from from Landing layer to bronze to Silver and after silver it was a selfservice platform someone wanted to create reports he was do it on its own and it was kind of challenging to simplify the complexity that sometimes is with the uh

11:36with the uh with the data platforms but in the end we've managed to deliver that as well as a a lot of trainings to the business users how the uh how the home platform Works how it was created and how to use it and how to maintain it that was that was really great project

12:00and another one may be the one that I'm currently working on unfortunately I cannot say like a lot about it but yeah a lot of data terabytes of data a lot of lines of code and a lot of work to do to make it fast scalable and and cheaper than it is right

12:24now yeah which which makes sense I and I think like you mentioned simple ATL maybe Lake flow will resolve this problem maybe maybe we'll see we'll see how it goes yeah depending on the source but you can just select the source and destination write some basic code transform the data it will be like put

12:46in MV or streaming tables and that you can leverage in U a dashboard or maybe Genie to do uh to do like uh to open it to business users yeah to be honest that's I think that this is the way where Databricks is going right now to simplify stuff

13:09because for some small or medium clients they don't need like white architecture they don't need a lot of lines of code all they want to do is to take data from point A and have them in point B and produce reports out of them with a little bit of Maintenance with a little bit of development that's all they need they don't need something

13:33complex yeah and that that makes sense the goal is to make to have things easier something to be done easier in short amount of time and to be able also have posib to update it and improve it without a lot of cost and I I had also other um other uh other questions

13:55like maybe can you tell us more about one of your uh like the most uh interesting features that you find in UC most interesting feature that I find in UC that's that's kind of a challenging question because all of them are really useful but maybe the data lineage is

14:19kind of a cool one because it lets you track what's happening with your data and maybe the second one is the uh whole integration with the unity catalog the ability to reach out to the unity catalog tables basically from

14:41everywhere from every other toour other other tool I find it cool because uh a lot of clients are worried about the vendor log that if they go with the Databricks then they would have to stick with a Databricks and thanks to the apis they can also connect their third party tools data processors Etc and they

15:04can still benefit from the Databricks from the unity catalog but they can use their own products and I find it find it kind of cool yeah yeah the UC open API brought a lot of a lot of exciting things and I they are still working on a bunch of other things like last time I discussed with the pm he told me

15:27like in the road map they will have possib to write from external engines to the manage tables because for now you can only read you can only read from manage table but you will have possibility to write which is good and then I think this is this was one of the main Blocker of people like not using the managed uh managed ones

15:49which are to be honest I I would still recommend people to go with the manage tables for many many reasons the first one for those who use Delta whenever we have a new feature you need to go and like add the table property and modify every single table you do have for example I don't know the list in vectors you should go to either have a way to

16:11list all the tables and then the script that would iterate and add like enable this delation vectors while if you're using manage tables those those features will be enabled automatically for you so we don't need to do any manual tuning same thing for if you have predictive optimization enabled like the uh the vacuum I know a lot of people tend to

16:33forget about this but to be honest you need to run it to save some storage cost same thing for the optimize we did add recently the analyze which collect stats to improve the performance when you're writing the table so I think those just and same thing we have more exciting things but we cannot mention them let's say maybe you need to wait for the next data I

16:55Summit yeah my my advice about the manag external tables would be if you don't need to reach out directly to the files on the storage then stick with manage otherwise you can experiment with external although having managed tables you can still reach out to them through the uh through the storage account but it's it's more

17:18Troublesome yeah because you have the uid you can't find the name of the name of the table in the P like with the with the managed tables yeah that yeah but since you're using Unity catalog then Unity catalog should be anyways your your first point of contact if you want to have a data not the storage accounts yeah yeah like the goal is to avoid H granting those permissions to

17:41the storage account to everyone and if either they want to upload some files they can just go through volume for example if they have the permissions and then it would be much easier and since it's just a button you don't need to have much experience to upload the data rather than going to AER which requires to be a little bit technical to go to the ADLs uh find the right path create the rep repository then upload your

18:05files then choose the permission so it's little bit can be a little bit cumbersome for uh maybe some business users yeah totally agree and Daniel before we rra up can you maybe if you can share some advice to folks because I know we have a lot of resources like if you want to get

18:27started do a data break but what would be maybe your entry points maybe some videos books or some links or any other thing you may think about of course Databricks Academy there's there's a lot of resources about the Databricks but I would start with

18:49the uh spark Basics and a books about spark as it's a as it's a fundamental to know spark once you're work working with Databricks watching the uh YouTube videos there's there's a lot of YouTube videos about how Databricks works and yeah the best way to learn

19:11Databricks in my opinion is to jumping on a project having the uh software engineering background and jumping on a project that's that's the fastest way of learning how the things works because you might know data breaks from the documentation but you won't have a hands-on experience and that's in my opinion the most important stuff when

19:35you want to be a data engineer engineer to have that experience to fail a couple of times but recover after a failure and to know what are the limitations and how to do some of the things that sounds good I think you summarize it super in very easy way

19:58thank you so much Daniel for for your time and looking forward to meeting you maybe uh at the dayani summit who knows who knows yeah thank you for your time and thank you for having me here