Data Lineage & Delta Sharing with MVP Jaco van Gelder

0:05hey folks today I am with yako he's very famous and well known on the LinkedIn Community as well as the data Community hio hey Yousef nice to meet you finally yeah we red so many times yeah well we finally have time to discuss I've been I

0:28think I've been following you for quite some time in U LinkedIn and then I was happy when I saw that you join the uh the MVP Community x uh beacons but if you can just uh introduce yourself to those who don't know you maybe yeah more than happy to so my name is yako I am uh I am originally Dutch living in the

0:51Netherlands in Brea a City close to einthoven for the people that don't know and I have been in data engineering for now close to 10 years um so I jumped into the Azure Cloud very early on I think 2016 or maybe even end of 2015 uh and that's when Databricks came into uh the first public Cloud as well

1:13so I I think I've been working with Databricks for over 8 years now starting somewhere end of 2016 we are now end of 2024 so yeah close to eight years um who am I personally so um uh I am a father as of recently uh became a father of Rafael thank you end of July uh so that

1:35put things into perspective I've always been really career driven and ambitious and um yeah focused on work but you know once you get a little little kid it all puts it into uh into perspective and you really get a different purpose but um I am married as well uh so that is uh uh me personally so professionally have

1:57been doing data Databricks and data engineering for the past eight years uh have done many implementations for many large organizations uh Banks Insurance funds um Pension funds right now at Ikea so that's a retail business obviously as everyone knows uh and I think during all of my uh Databricks projects I have worked with Databricks themselves so

2:20with their solution Architects their delivery solution Architects and um yeah I've really um gotten a bond with them um I even call them my friends now so uh and as of this year I am also an MVP so I think that is me professionally congrats and we can your baby crying oh can you oh man I I not a big deal I had hoped it wasn't uh herble

2:45but yeah that'll make it fun is it loud or you know no no no no no but we know that you you're a daddy and congrats yeah I can send my wife a text message to try and keep him a little quiet but uh no no no all no no no let Rafael Express himself exactly okay I appreciate that so you said you've been using data brakes for like almost

3:09eight years almost eight years and I remember I used it uh during my master degree and back in that time I think we had only the ml spark and that's it just just interactive clusters some ML and some notebooks yeah and jobs exactly job and jobs but you could you but you could orchestrate things you need to call like

3:31a notebook from a notebook from a notebook that's how you orchestrate the good old days yeah yeah so what do you think regarding the changes like like the U the evolution of the platform for the last couple eight years like or let's say just four years like you see it change tremendously and what's your opinion and what's your favorite part of

3:55the changes right so that is an excellent question um so as you as you mentioned and I think back in the day many people used Databricks at least in Azure just as a way to run some python code in the cloud so I remember back in the day you had Azure functions but it wasn't able to run python yet for example and people would just schedule datab bricks jobs or datab bricks

4:17notebooks from an orchestrator data Factory airf flower whatever uh and that is what people use dat RS for they didn't at least even I can speak for myself and say that in 2016 I didn't even know what spark was I just used data R to run some python code in the cloud now I know that is something very expensive to do you would definitely not use it that way uh but I think

4:40especially here in Europe I have not seen many organizations using spark I think it is also because data volumes are not that that big or that high uh at least not back in the day um so yeah it has evolved you like tremendously um I think the biggest change have been that obviously data R

5:02has gotten a lot of venture capital injections right so the platform grew both the platform and the company grew uh enormously uh but what I really like seeing in the last couple of years is that they try to create this holistic view of what a data platform should be so not just running python code or SQL code but also adding governance adding ml adding AI adding uh model serving and

5:26things like that to really make it a data intelligence platform as call it or as data rck calls it now I I think that is the the biggest change and the nicest thing I've seen happening so it so going from a platform where people would run python codes to a full-fledged data intelligence platform that it is today um and right now you don't have to

5:49go outside of data rck to do anything back in the day data rck was just something you would add to your platform or add to your uh data ecosystem but now it is you can just use data rck and that's that's it so you have all the tools at hand basically and that's what I really like and what I've yeah what I really appreciate has happened in the last couple of years yeah that's uh you've summarized

6:12it very well and like if you can uh like you mention that you did so many migrations and can you all cast through maybe some best practices for folks who are like say completely to data braks and the ETL part what would you recommend them to start with and what are some bre

6:36practices so I have done a lot of migrations mainly from o2p systems like SQL server or Oracle database towards the Azure Cloud including Databricks um and obviously um so those systems are really heavy on SQL right on um on SQL code um and migrating that to Databricks even though spark SQL is Rich at

7:00the moment it's not a one-on-one copy you need to do a lot of well not a lot I think I think 90% of Oracle SQL or or transact SQL will work inside Databricks but there's a couple of things a couple of edge cases that don't work uh and for that I think you need to do um yeah you need to build specific logic you mainly in python or build custom

7:22udfs to fill in that Gap that will that yeah that you cannot migrate from the old SQL environments um so the best practices that I would have is it is never so if you migrate it is never lift and shift people want to have a quick migration and think hey let's do a lift and shift doesn't work like that um I am a really big advocate

7:45of um maybe not completely rebuilding it but refactoring it it at least to work with the cloud because also coming from on premise to going to the cloud there are two different worlds you cannot expect goat from on premise to perform to run really well in the cloud you need to do some sort of refactoring so that would be my advice is don't lift and shift but yeah really design for the

8:07cloud when you do a migration um and then when you go into Data rig so um and I know there's a lot of discussions going here and there about this but um I think the easiest way to get data out to get value out of data quickly is by using notebooks obviously if you have a team that is extremely experienced in um in in creating python uh software for example you can you can use Python

8:30packages uh but I think to get value out of your data quickly and when migrating from old SQL environments I think notebooks are the way to go uh and yeah it can be as simple as just scheduling notebooks inside jobs or um creating um the I think there's stored procedures now I mean the old systems heavily rely on stored procedures for ETL I think you

8:52can do most of that in Databricks I think that would be those would be my uh I don't know how many cents eight cents or something yeah yeah you're uh you're you're completely right sword procedures are not there yet we have secret scripting store procedures will be out almost next year yeah almost there almost there yeah but I mean like like I think so people

9:17overestimate they look at the word store procedures but what I really see people do uh migrating store procedures from transact SQL for example towards data rig is just just re rewrite the SQL SQL code inside notebooks and you have a you have a store procedure pretty much I mean it's not called a store procedure but you can use the notebook with the old SQL code as a store procedure if you like um you will get the same results

9:41wouldn't say it it is the it is the best practice but it is definitely a way to go uh that makes sense and I just have some like few questions like can you tell us more about let's say one of the most exciting project you work you worked on like the name of the company like just tell us more about yeah so I

10:03won't mention the name of the company but I think people can can can can guess based on uh on the story um so yeah I have done a lot of really interesting Pro projects in in the past but I would say my current project is really the most fun one uh because it it the scope of the project is huge it is a it is a global scope so we are building a datab bricks platform for this company won't

10:25mention the name but if you look at my LinkedIn you'll see what company it is but we are sharing sharing data with all of the companies um stores business units across the globe um and not all of these um uh let's call them data consumers not all of them are on Databricks for example some of them are on gcp and are using big query some of them are just using on

10:48premise solutions to process their data so the challenge there really is to share data in a way um where they can receive it but we're somehow still in control so we came up to do uh we we work with the platform team inside the company to set up Dela sharing in a secure way so we cannot only share with data RIS consumers but also with um um bigquery consumers or just random on

11:12premise consumers um but also also datab brick consumers obviously we're we're running on Unity catalog so it becomes very easy to share data with other um other people inside the company and I think the fun part is the thing that makes it a lot of fun is is the skill like I mentioned it is a Global skills so I'm working with people from all over the world um USA China Dubai Spain

11:36wherever um and also the volume of the data obviously is huge because we're extracting A system that is used by 250,000 co-workers worldwide um so also the volume challenges are there so making sure that you extract and and and transform data at skill um data breakes performs really well but you really have to turn and

11:58tune some of the knobs to make a workload like this perform um so that is what I really like about the assignment but I've done a lot of assignments in the past also for banks leading Unity catalog migrations that I I thought were really a lot of fun setting up Unity catalog for example and and collaborating with the guys on GitHub in the open source Community um of the

12:20terraform provider I must add not the unity catalog open source that we currently have but uh I did some uh some some some contributing to the um to the terraform provider that uh Databricks has that has been a lot of fun um yeah I think I think those are my my most fun achievements uh to date okay

12:43and uh you mentioned like a lot of parts on Unity catalog so many things can you tell us more about let's say if you had to choose one favorite feature oh I have a very favorite feature from Unity catalog and that is the the data lineage component that comes with it um so I've worked a lot in

13:07uh in the finance industry and in fi in the finance industry there's a lot of regulations and Audi things uh audits happening right so at the end of the year you have a big four uh Consulting uh auditing firm come in to do a couple of um audits um and yeah it becomes really hard if you don't have data lineage there um so for example if you you have

13:29a bank that is reporting its yearly figures to their um to their um um yeah to the people investing into the bank right um and you're not 100% sure that the figures that you're showing them are right um that I mean that's what an auditor does when he comes in he he he makes he checks to see if the figures

13:51that are being presented to the shareholders that was the name I was looking the word I was looking for are correct um so I yeah data linear for sure is my favorite feature because of the uh the auditing purposes that that it brings I think it is a a huge game Cher I've never seen it work this well uh meaning on like column level lineage and things like that in any other

14:14product and that is why I really like that as my favorite favorite feature of unity catalog 100% but there's other ones as well like the the table comments the column comments that you can add metadata to your uh to your to your I really like those as well and they also help with the AI uh products like Genie so Genie looks at the table

14:36comments and uh column comments actually to u to come up with its response uh I think that is just brilliant so I think those are my two two favorite features lineage and U the column in table comments that you can set yeah that was a like super interesting answer with bunch of with

14:58the bunch of things you mentioned and uh like besides like like let's say what you like about data braas but is there anything maybe you would have loved to see in the platform that's a really good question um I have to dig deep to answer that one

15:20um what what would I like to see in the platform so there's a couple of things that I think are still a bit lacking in Databricks especially now that the wider public as start is starting to use the platform so I think you need to be quite of a technical person to um to use Databricks and I think the serverless offering greatly helps in reducing that um but for example if

15:43you're just running usual small workloads uh on job clusters for example that still run on spark and you run into some errors you get a huge Java Trace log as an error and I think many best experienced people have no idea how to debug those uh so I think data rcks and uh this has been like this since day one

16:06so I'm I'm still a bit I'm I don't really understand why datab bricks didn't yet uh make a move on improving those error messages I think servess error messages are a lot better by the way but just this the classic U jvm error messages that are still in the platform I think they can be made better to really help the general maybe less little less skillful but public a little

16:29more I I think they did actually uh add or simplify the error stack for DBQ yeah for DB SQL 100% yeah because I mean that was the goal because DB SQL is mainly for SQL analyst why you're using spark your data engineer so technically you would uh have a better experience

16:51when it comes to troubleshooting those kind of errors but I completely understand and that that makes sense but like the first or the change was done first for DB SQL which would make make sense and I I think they will still keep working on improving the uh the um the stack era yeah I'm sure they will

17:13100% yeah I hope so at least that I think that would really help the little little less skillful people and uh like we talked about best practices but if what would be your advice to the uh uh people who just got their diploma

17:35like how they can get started because like you've seen like you have so many platforms so many languages so many Technologies yeah and sometimes they can get lost overwhelmed with all those kind of information so what would be your advice yeah so very good question so uh back when I started in

17:572015 um the landscape of data engineering tools was already quite big and there's this image that you see grow every year where the tools become more and more and yeah and and so it becomes a bit of a problem uh especially for newcomers to choose what to specialize in um but really my advice would be um choose basically you have to

18:21choose you cannot learn all of the tools um and what you should not forget is the concepts and theun foundations of all of the tools they overlap large like on a really large amount so if you know Databricks you can do a bit of snowflake obviously you cannot maybe know all of the details of both platforms um very extensively but I

18:44think the bottom line or my main advice would be just choose a tool and specialize in that one while maintaining or or scaling up on the concepts that allow you to also use other tools for example I chose to specialize in dat bricks that doesn't mean that I'm not able to use snowflake or big query or whatever um I can because I know the concepts across these platforms somehow

19:07overlap or what overlap in one way or another um but yeah I'm I'm not as good in Snowflake as I am in Databricks obviously I have my snowflake foundational certification but I really chose to specialize in Databricks because I because I chose I think you just have to choose choose a choose a platform choose a stack and then just specialize in that one and then if you want to go

19:31t-shaped don't don't uh horizontally scale or scale up on other tools but scale up on Concepts like networking uh infrastructur R code devops those types of things uh don't go learn all the tools that would not be a good advice I think that would wrap it up yeah I think that's uh that's that's

19:54that was amazing comments and um do you have anything else to add before we close um I don't actually know uh unless you have any questions left then no I I don't have anything to I'm really happy that I've become an MVP um because it really allowed me to um to meet a lot of very good and also like-minded people so

20:17I I must say I really really like the MVP program and I really appreciate for appreciate data rcks for putting so much effort and attention into it yeah that was an amazing program used to be called before beacons and now has been uh revamped and called now databas MVP I think MVP is a better name than Beacon yeah yeah yeah yeah because now

20:39you can link it to data breakes which is and to be honest guys you're doing an amaz show like I've seen I can see your posts like posts of like like a lot of people like uh uh yeah I love hos the man with your orange head yes yeah exactly and not only we have

21:01bunch bunch of all them are great I see now the post floating literally every day and you're doing an amazing job thank you Yousef I really appreciate that thank you for being my my guest and see you soon thank you so much see you later byee