Data Migration & Unity Catalog with Databricks MVP Julia Førde

0:06hey team today we are starting a new concept we will be interviewing several data breaks MVPs because if you don't know the program has been rebranded it used to be called beacons now they have we can say a real name data breaks MVPs and I have the absolute pleasure to have J J with me as

0:31a first guest hi Julia hello and thank you for having me it's a really cool to be here I've been following the channel for a while so it's going to be fun amazing and I used to know you as well even before it was announced uh you have been announced as data R MVP and

0:53can you tell us more about you about your background yeah sure uh I've been working in the data space uh fresh out of University 5 and a half years ago so and i' been working mainly in the DAT house data warehousing world so I actually my first years I worked mostly

1:15on premise uh within the Microsoft stack I did a lot of data warehousing uses like SQL Server integration Services analysis services for models and some powerbi on top though that's not my strongest uh suit or interest but I have done that as well and then as the client started moving more to the cloud especially the last years I moved along

1:39with them uh they have some experience like aure data Factory and that's those types of solution but then uh I basically thought that my current customer at the time would benefit from uh this platform called Data breaks uh so I suggested that we looked into it went to a world Tour event with data

2:01breaks and we figured out that this was the right place for them to be uh so yeah that's my background so yeah you have been using data brakes for the last couple two years two three years right no not that long even like closer to two then um so um yeah it's been a very intense Journey

2:26let's say that um and I'm mostly like focused on uh The Lakehouse concept and doing migration U which is quite natural as I come from like the on premises stack and there are a lot of migrations going on so that's kind of my focus area yeah and knowing your background I think I think you started at the um the

2:49best time ever because uh as you can see like data break SQL data whereare in offering started getting bigger and bigger with a lot of improvements lot of features and it's mainly started let's say I mean that was three years ago but we seen like u a like a huge adoption um two years ago and I think so you're part

3:11of this trend that started and benefit from those cool stuff like Photon and all the other exciting features that that were introduced with DB with DB SQL but but before I'm curious to know like uh how when the data breaks reach reach out to you to announce that you have been nominated as an MVP oh this is

3:35always the interesting questions I've had the last especially the last months like all these people asking of course like how did you do it uh how did it happen uh how did you like go through all the steps and to be honest I didn't really I I never imagined actually that I would be a part of this type of

3:59program I just seen the people being part of the program and think okay these guys they are like next level um so to be honest I haven't actually focused on it at all and what I have done is that I've spent a lot of time going around in events uh in Databricks meetups I've also this year really started like my

4:23speaking Journey so I do more like presentations in communities talking about Databricks and that has kind of w widened my network of fellow data nerds which is my favorite type of people and so I've done a lot of speaking and recently started being more active on LinkedIn and and then to be honest the

4:46main reason is that I've been working quite closely with Databricks in the nordics so I've been working with some amazing guys like account exe Executives and some solutions Architects which are covering like the Norwegian market so in the nordics they have a um Databricks office in Sweden which also covers uh

5:10Norway Norway where I'm based uh and that company basically I've been working quite closely with them on customers and like talking a lot about the tech with these people and it was probably through that combination of uh starting my speaking Journey being being very active in the uh community in Oslo and also

5:35then actually working with them on customers and kind of evolving customers and helping them on the journey so I haven't really been focused on getting into a program I've been just doing my stuff and then it happened so that's kind of the yeah probably not the answer people were imagining but uh that's what

5:57happened yeah I mean that's it just do what you what what you want to do like keep reading keep learning keep posting and then it will it will come to you naturally and um and since you mentioned that you started your uh like communication or public speaking journey I guess we will

6:19see you probably applying for talk at di oh no that fresh well we'll see we'll see I'm not gonna comment on that it was just opened I have heard so we'll see what happens but uh at least I enjoy speaking at other events and then we'll see if that will trust me you you should do it you should do it it's it's an

6:41amazing experience I uh I had a chance to be uh to present with the with the fellow friend uh matu Powers uh one of the committer I have on spark we had a session on Delta Lake and we had like a full room like session was like getting not get inside but best Delta like features and trust me we had full room

7:04like 180 people others were like literally sitting uh on the floor and they stayed for like 1 hour and then we continued the discussion outside because they kicked out from the room because they had another another toll and like the energy The Vibes it's just amazing

7:26uh so if you have this opportunity just go yeah I actually went there last year as but just as an attendee so I was really lucky to be able to go and yeah that I think that event and the summit that was kind of what brought my motivation for Databricks like even I've really like amped up on like

7:49reading and staying up to date because well we both know that's probably a full-time job just trying to keep on top of what's being released and the it and the amount of stuff that was announced was just like yeah it blew my mind it was uh really cool and I expected it to slow down a bit after the summit but then it really didn't no no no actually it's always so

8:15just you like I I tend to go almost every year just to keep up this energy because whenever there whenever I go I am full of energy even even if I'm jetlagged uh like lack of sleep uh waking up very early going like sleeping very late but still the energy uh is there meeting people exchanging the

8:37Keeping Up with the product and and trust me Julia like uh I went on a PTO for two weeks when I uh went back uh I opened the platform a lot of things changed like how to create clusters and stuff was like so are they telling us maybe like should I stop taking p because

9:00whenever I do whenever I I live sze for like two weeks a lot of things change and this is the really the power of data you are never bored if you want to learn things you will keep learning like for ages and I think this is this is and this is a perfect intro to tell us since you're you're on a data warehousing World for for data analyst like there

9:23are new joiners people that are just getting started with Databricks what would you advise them like the res resources to track or what to do to maybe increase their knowledge and not be afraid from using the platform yeah so I think one of the what you just mentioned is some of like the really amazing stuff about data brecks but it's

9:45also the thing like the flexibility of it and how fast it kind of evolves almost is also the biggest challenge when working with Databricks because you have all these possibilities and you have you can basically do anything anything you want in there so I think one of the main reasons is just really set some goals on what you want to achieve and what you want to focus on uh

10:09at any given time because if you just basically unleash all your developers and tell them to go nuts then they probably can so it's kind of um and I think this comes from like my data warehousing background where it's very much about kind of structure having some common frame work and some common ways

10:31of working to make sure that you kind of keep control of your solution and that the data within your solution of is like of a certain quality so I think really taking the time to decide on what you want to achieve and in what order you want to achieve it uh is a good thing and also maybe if you have the

10:54opportunity to spend some time to actually develop some frame works or some naming convention like anything that can like provide your team with a bit of uh predictability is also nice especially if you're in the data warehousing World um I think that one of the greatest strength of a team if is if

11:19your solution is a bit developer independent you don't want to end up with a solution where it looks completely different based on who has written the code for example right because if that would be your single point of failure if something happens to that person if that person quits or if that person gets sick uh then it's hard

11:43kind of for other peoples to take over so I think kind of having this uh a lot of people that meet me they expect me to be like this uh very like do what you want not that strict type of person but then I come from like these data warehousing principles which I also think we have to find like the middle ground between going nuts and being too

12:07strict so I think kind of finding some common ground is probably one of and have a plan which probably will be one of my best advice that that makes sense that makes sense because else you can get lost and and you don't want to have also this uh this loing and I also seen like you mentioned first during the beginning of the discussion you are handling

12:31migrations and I've seen that some customers sometimes can be afraid from migrations because it's a change of uh uh like you're changing the system which means you're changing way of doing things and sometimes they tend to some some people or some companies they tend to reproduce what they had in the old

12:54system rather than benefiting from the openness or the scalability or the Liberty to do anything they want no they tend to reproduce what they have so how do you how do you tackle those challenges how do you convince them that this is the best way to do things what's your strategy so first of all of course like

13:17having this initial project that we have kind of seen like this use case that it works is of course like the B biggest Advantage but I think it's also like make I think a lot of people are saying things like ah the data warehousing is dead the data warehousing is ancient and I think that's a misconception right because I don't

13:39think it's true I think the data warehousing isn't dead we have the lake house but it's not doesn't mean that the data warehouse as a concept is dead it's just a new form I see I try to see the lake house as like the newer and expanded version of the data warehouse because in theory the problem you you

14:02still need to save uh solve the exact same problems as you did in the warehouse right you have a bunch of data they're coming from a bunch of sources and you want to apply a lot of business logic cleaning structure and then end up in some reports or some machine learning models and that still applies it's just

14:24that you can do that which structures data and you can also do those types of things with unstructured data you can have machine learning models in the same platform so it's more like a lake plus Plus+ version of the warehouse so I think also a lot of people get scared because these types of things are thrown around and like these opinions um so I

14:47think that's part of it so I think actually what data braas does and actually say they are actually using the world words warehousing like they're saying data breaks warehouse and I think that's a very very uh good thing because that's where people are coming from I think a lot of people that are they're used to on premise they've been there for 20 years they've been there for 30

15:09years they know a lot of good stuff about data modeling about structuring the data but then they get this feeling that the lake housee is something completely different um and yeah is different you're working with file you have new file uh you have new you're working with more with files you have new formats uh you can also include new

15:31Lang uh so I think a lot of people kind of get scared by the fact that they think it's something it's more different than they it's not that different that they think yes you have to uh have some you can add some new cool use cases and you can still continue using in many case your the same language so I think the key is

15:54convincing these experts that have been building the solutions for decades and kind of understanding that getting them on board and making them feel comfortable in the new uh platform is kind of essential to building good Solutions so I think you need people with the cloud experience and you need people with like the old uh warehouse

16:17and on premises uh mindset so I think kind of making sure that both these two teams feel comfortable is probably one of the most important things to be able to convince a b business to yeah do the migration that that makes sense and I also think that it's also blocking them

16:40sometimes like also when they move from those uh data on premise dataware houses where they have this lock in because the data the data is in specific format and then when they go to uh for example data break they need to choose like uh between the storage format whether it's going to go for Apache hoodi Iceberg or Delta Lake then again why how and then

17:05other questions start emerging so it it make it it makes sense but I think you also posted some like lately something about uniform where we did explain like the advantage of uh of using uniform so can you tell us more about your your experience with the with uniform so basically I just think it's

17:27very cool as a concept uh I think when Databricks bought acquisitioned Tabler a lot of people the days before in the summit were also like what does this mean some people are skeptical they were like does this mean that they the bricks are is actually going to do a vendor lucky now right because I can understand that some people might have gotten those

17:50ideas because they they felt like they bought the format even though it's open sourced of course what all the people that created that so I think a lot lot of people kind of ex was afraid of that and then when data breaks went on stage on the keynote and they just open source Unity catalog it was like a very clear uh saying to say like well we're

18:14not going to Wi lock in anything we're just going to make it even easier for you so I think the the IDE data breaks now has by making Iceberg and um Delta so uh similar that you actually don't have to choose that's a game Cher and uh by doing that they actually say

18:36they don't say you have to use Databricks what they're saying is we just want you to have more freedom so that choice isn't forcing you in any direction so I think that concept is really cool that they're really following this open sourcing uh mindset yeah that's with Delta you can

18:58go but like uh you can go to H to hoodie and also Apache Iceberg so they're adding more and more uh integ integration and hopefully uh you connect is coming that one is I'm quite excited about as well yeah yeah so L flow connect that would be also something a

19:20game changer because like for you I'm not saying you don't know right python but for for you as a data analyst data data data warehousing expert you can just go click select any system I don't know can be a SQL Server you have the CDC enabled by default and you can just ingest data easily from from there yeah

19:43I'm working a lot with like Dynamic 365 in Microsoft it's a it's a bit of a pain files to work with so they have said that they're getting a plug into that one so that's probably what I'm most excited about yes yes Dynamic from Microsoft the Microsoft biggest Microsoft country probably like in in Norway it's um they

20:08have close to 100% of the market basically so there's a lot of uh Microsoft on in my daily work yeah yeah that's that's that's pretty that's pretty much good and um um tell us more about like um the advantage of the uh the Databricks MVP's Community like uh tell us more

20:33about the benefits and what you have been learning so far since you joined the program yeah so first of all uh I probably get to go to this the summit the second year in a row so that's super exciting so I'm looking forward already I they have dropped the dates now so I should just like order my tickets so my

20:57company doesn't change their minds and then um the other thing is that like we have meet monthly webinars where like the product teams like Dev real teams from Databricks actually come in and demo us new stuff for things that they have on the broad map where we can ask them questions uh we have like this slack

21:19Channel with quite a lot of activity and there's a lot of databas employees in that so basically if I I had this question about the system tables and I had some like um things about that which I was wondering about and like were thinking about and then I just said in the section and like hey guys is anyone

21:43working with this Databricks and then a guy got tagged Greg kski and he I just messaged him yeah basically and then he messaged me and he was like okay let's set up a meeting so I can understand the use case and I think that's probably one of the coolest thing like getting so close to the people actually behind the

22:05Technologies and being able to kind of understand but also give feedback in what's not working that well what is working well uh and of course it's nice to know a bit things before everyone else is always nice to be prepared yeah getting a sneak peek like when you watch your favorite series and

22:27then you watch like the sneak peek from the next episode and you you always want to want more but then then of course it might be annoying as well right because you just want to start right away um and then but yeah I think in general also Databricks is a company that is quite out there with the customers even though you're not an MVP or anything like before that as well I felt like every

22:50time we had a problem it was very uh easy to get hold of data break so as a company I also think that they're uh quite in front there but of course it's a bit easier even easier now so yeah and then I get some swag which is always nice right yeah I had this one as well yeah exactly so uh

23:14yeah and and one more thing before we close what is can you tell us more about the your the favorite let's say use case or project you did work on maybe for the last couple two years oh yeah uh I've done a migration to the data platform um um I'm not going to say

23:37from what but uh basically I did a migration to the platform and we had a massive reduction in costs uh per load so we are now actually able to run the whole data warehouse like we run it every hour on the hour so it's 204 uh runs a day of the data so the users get hourly updated data which is really nice

23:59we've seen like the adoption of powerbi also kind of skyrocketing because now the business is more able to provide this um fresher data so I think that's a really cool project um and of course uh just having Unity catalog actually enabled in a platform is

24:23Magic yeah that one is um yeah I don't understand if you can do anything without your L now to be honest because it has so much good stuff yeah I was wondering I was I was wondering or asking myself the same question on you literally need Unity catalog because else you just have your data then when

24:46you see like the amount of things that Unity is bringing like lineage uh system tables uh the governance discoverability uh you have some insights like you can click on any table you can see how many people quered the table who did most like most frequent users with which table this one has been

25:09joined you can Define primary key forign key for tables you can have this automatic uh comments a lot of things like I can I can keep talking for for hours I'm really excited about the this uh which they announced on the summit the attributes based access controls that's also a game changer when it comes to the stuff people really don't want to

25:31do in data engineering project is like masking sensitive information and then you probably did it wrong and it's way to manual so that one is also really cool oh oh yes yes and especially they they they GNA they going to merge it with lake house monitoring to have this automatic classification which means you know this column is pii so it's

25:53automatically tucked pii and then based on this you can put your rule like uh only leaders can view this uh this this Pi except for example Julia and also the things that I think find many very interesting is the time based access like you can grant you the permission for like 20 days and after this the

26:15permission has revoked because now and it's avoid having this issues when you grant someone permission then you forgot to remove the permission because let's be honest that happens like if yours it's easy to forget actually like if it's a big project or if you have a million things to do this revoke of access it can be forgotten and it will

26:38be in many cases so I think that's also really cool especially like if you have people that are just in there to do one specific task or need something for one report yeah so I think it's good yeah the that makes sense I think that's pretty much all I think you have covered a lot of things you have shared several insight and we're going to put your

27:00LinkedIn on the comment or the description of the recording so make sure to follow Julia for more information about data breaks and why not watching her session at dice next year thank you zulia thank you