Databricks Platform Evolution: 2017-2026 Roadmap & Learning Path

0:05Hey, welcome back to the MVP series. Uh, today I have the pleasure to host Andrew Sitz. Hi Andrew. Hey, how's it going? All good. All good. I'm so happy to have you today on the on the talk. Um, can you introduce yourself to the community? Sure. Absolutely. I'm Andrew Sitz. I'm based out of De Moine, Iowa. So, I'm

0:29kind of far from a lot of the customers I work with. Um, but I got my start um working in business intelligence with a Midwestern ger. I originally have a management background and used a lot of operational knowledge to get into like business intelligence which then eventually branched into more uh data

0:53engineering and analytics engineering. So yeah, I turned from business into more of an IT role and at at times I make uh appearances in bigger cities like uh the data and AI summit this year. Oh, cool. So we're going to meet for now I haven't had the opportunity to go to Iowa, but

1:15maybe now I have maybe I will reach out to you if I need some tips like what to visit there. Yeah, not the biggest tourist destination in the world, but there is some stuff. Amazing. Uh, so I guess have you applied for a talk at Data Summit or you're just going there for because you're an MVP? I am going to submit uh for the MVP gallery. Okay.

1:37Yeah. Didn't get a spot with the normal talks, which is totally fine. Um I have spoken there before but yeah really looking forward to meeting a whole bunch of people in the community. Yeah the data summit is just an amazing opportunity to connect with so many people and uh I think this year I think last year uh they had I don't know

2:0015,000 people and now this year I think it's even more. So it's just crazy. But yeah, it's a very good opportunity. As you mentioned, you're going to have uh the um community booth where you can like uh uh deliver some talks and so it's going to be an amazing opportunity for uh uh for you Andrew and since you're talking about the uh data AI

2:24summit. So um when was like the first time you attended the data and summit and when did you had your first uh Databricks experience? So the I've just attended the data and AI summit in 2023. Um I've attended it virtually every other year for the

2:47last 3 4 years but I got a start working with Databricks is like a backend piece of architecture um on Azure and at the time I would say it was really just centric on back of house pipelines and then we would serve uh data in other tools espec on Azure at

3:10the time. Uh it was highly oriented towards serving more on like a SQL server or a uh Synapse type experience. But then I got a start with a partner that was really dedicated uh to Databricks um after having just made some bets on it like for my own

3:34personal career. uh just seeing that it was up and coming and went and got some certifications and really started to pay heavy attention to it and it just became a whole realm of its own. It seems that you you made the right bet and good good for you and and how did you end up uh being an MVP? So yeah, working in the partner channel,

3:58I had the opportunity to work alongside uh Databricks in some really planetary scale solutions and uh got exposure in the community that way. Um, and just I I was invited to be a part of the Champions program and went and

4:21co-founded a firm with another MVP, uh, Josh Adams that you'll have a talk with sometime else. Um, and we took off on an endeavor to make our own firm and were able to get into the MVP program uh, during that stage. Uh I wouldn't attribute it to like one

4:43particular event per se and getting in more so just a series or continued stream of working together with with Databricks on a number of different activities. I believe you you seem to be very lucky because you did a lot of bets and they all turned up good bets. Good. That that's that's awesome. And you

5:05mentioned like you worked on several project maybe. Can you tell us like maybe like one of one exciting project you worked on lately if if you can of course. Yeah. Um I've recently I worked with a client that is in the cryptocurrency space uh getting involved onto uh Databricks and unity catalog. So

5:28it's more of a foundational experience. But yeah, I've had the opportunity to work on things with some enormous labels or just some really impactful experiences with the customer. Uh ranging from uh helping to automate migrations. uh as like the firm Apoint digital. I work with we have a

5:53uh an alternative solution to um a as like a migration accelerator uh for with uh both automated and like AI in the loop type of an approach. So I've been heavily involved in in that side of like just planning the initial stakes in the ground of uh your lakehouse and then

6:18also you know even getting really in depth into other areas like using DBT on Databricks uh quite a few different use cases. um really a fanatic of the workflows capability inside the tool and how flexible and extensible it's been with tons of customers that I've

6:40worked with. that that that's really uh really amazing and I believe like what can you uh give um to aspiring data engineers who are just getting started with I don't know in data first and then uh in Databricks um taking the time to just even walk around if you work in a

7:02company and it when you're at events if you work virtually or just getting the opportunity to learn as much as you poss possibly can about the business that you work in. Um, and having just relationships with stakeholders builds trust where you you can get a shot at building them a solution even if they're

7:25not able to really envision it beforehand. Uh, a lot of the process in my experience has been internal selling uh in the organization that you're in. So being highly connected internally and finding ways to be reputable by really knowing the business or knowing ways that you could impact the business

7:48properly, not just uh blanketing maybe a solution that that won't have an ROI. I think that's extremely beneficial for anybody to know in in this space. Um it requires a level of extraversion. I think that may not come as naturally to some of some people that are really gifted in the technical

8:10space, but it's highly worth it. But then I think you were pointing towards what I'd give recommendations on for somebody maybe looking more specifically at Databricks, right? Um, I would say that I still think it's highly important to start with the open source

8:32elements of Databricks on what really has laid it out and cemented it is a humongous player in the space. Um, to just have a found to have an a baseline understanding of really what's happening. um get a feel for

8:55uh turning a number of the a number of knobs um so that if when you start to use uh abstractions in the product if you choose to do so that uh you'll know like in autoloader what some of the properties are like a max files per trigger uh things like that I I

9:17think uh trying to build solution just with the foundational building blocks. uh helps you to learn a lot and then it also helps you to appreciate a lot of the things uh that can make your life simpler such as you know predictive optimization now on Databricks when you

9:39used to have to completely handle those types of things on your own or even dealing with 100% unmanaged tables back in the hive meta store day. Yeah, I think there's an aspect of like it's kind of difficult to learn that foundation now. Um because people like start their knowledge using

10:03things that are that have been built upon for a decade now. Uh so if you do get even the chance to talk to somebody that has been working like in this platform for quite a while, I would take the time to listen uh to their perspective on all the advancements that have happened because uh chances are

10:26they they had to solve for these same issues uh quite some time ago and can probably talk about you know how is life a lot easier now? Um, how is that problem even any different or is it just like not even a problem anymore? For example, like having to know partitioning columns up front versus

10:50like being able to take advantage of automated uh clustering and liquid Yeah, liquid clustering. Yeah. And I think I think we can do like a basic comparison. I remember I used the platform for the first time in I think 2017 or 18. I don't remember and back in that day uh we had only I think uh spark the so we

11:13have notebooks where you can run your spark and I think um jobs it was like you to in order to um to orchestrate your job you need to reference a notebook in another run notebook and uh and so on and then later like they added like workflows they added DB DBS SQL

11:35then they added feature store. Of course, the AI part I did not mention but was there since the since the beginning then they added unity catalog then and now if like if you see like all the features like I don't know the uh the genai stuff the AI functions uh I don't know the apps just like the platform is now so big covering so many

11:58things like a dashboard experience beanie so yeah the ramp up ramp ramping time would be I I think uh bit longer but I think this is the platform is now targeting or simplifying task for a lot of people. In the past they should they had to learn a lot of things to be able to orchestrate or to or to have a

12:20data pro project. Now they have everything like they can stay within the same platform build their I don't know use VS code to code then push the code to the repo run it on work on run it on Databricks orchestrated with workflows then use those KPI to I don't know to train a model or maybe uh to build dashboards on on AI on gen on aibbi or

12:44maybe genie to ask some questions and maybe deploy those metrics in a Databricks app while I mean for in the past I three or four years ago you would have required at least four or five stacks which means having to learn more uh more platform more technologies. I think this is also like the platform is getting bigger also covering more uh use cases

13:06and again I think I think having discussed with many many customers some people only use the engineering like data engineering side and others use only the data warehousing side you don't need to learn everything related to the platform because you can't simply you can't learn everything I mean we're humans like it will take time of course if you're passionate then you you can spend time doing doing this I don't know

13:30What what do you think, Andrew? Yeah, I agree with that. That you just pick especially the area you're the most passionate about um if you get to work in that area that you pick an area and really dive deep into it. But it is it it's not going to be possible to follow all aspects of the platform because it's quite a number of

13:54different career routes and to do all those well. I haven't haven't seen um people be able to keep up with all of the different areas, but I I think uh I know what those areas are for myself for sure. Yeah. Yeah. I mean and I think the only the only opportunity I had to be

14:16able to keep u myself updated is like either follow like the uh the MVPs like the blogs they are releasing I don't know kind of getting started with the Databricks apps uh how to do model serving so we have this we have also the blog post in I don't know if people are aware of it but we have lot of technical blog posts in the datab bricks community

14:40and of course we have you have events like Andrew mentioned like the data and AI summit where you can like you have the possibility to attend in person if you are in the US or able to travel to San Francisco or you can register for free for the online webinar and I think later in maybe September or October all the uh uh breakout sessions will be

15:02available in the uh Databricks YouTube channel and also I I remember like every year after the summit I spent at least two or three weeks watching all those recordings because we have so many stuff released and so many cool stuff. So we want to just like hey let's watch this recording and then let's go and apply let's use AI functions and okay let's go try it like we have UC open source okay

15:24let's go try it with I don't know DuckDB or any other tool we have this feature and it's like a way to keep yourself updated and also be aware of everything coming into the market because the data is not only about Databricks but also about the data and AI market with all the partners you have uh uh I forgot how do you call it like the lobby where we have all the vendors

15:48displaying their technologies. My mind is failing right now. I cannot remember the name, but it's I think it's a very very uh good good opportunity. Plus, of course, the meetups. I don't know if you have any meetups in the city where you do live, Andrew. I think also it's a very good opportunity like the meetups or user community group. Yeah, the meetups for me are

16:13the closest city that has one is Kansas City and that's about three hours away, but I um am planning to even attend that group at the end of this month, April. But uh I know there are even local meetup groups uh that I go to that are almost always present even if there

16:36isn't a Databricks meetup in your city. And I've talked to so many people even just locally about uh what are the types of things that they're doing and you'd be surprised just how many conversations you could have about if you want to have specific topics like Databricks there's probably going to be people looking to have those conversations at those meetup groups. So

16:59I'd give a shout out to like the de mo data and analytics meetup group that I go to locally here. Yeah. So you all you have a lot of lot of lot of opportunities and I may have a tough question for you. Uh if you can choose maybe your top three features within the platform, what would they what would be those features in the areas inside of the product? Um I would say I am most

17:26passionate about the data warehousing product getting built out. I'm super excited about uh not just like the SQL capability but SQL and serverless and meeting BI workloads for quite some time. Uh I am really really chomping at the bit to test out the semantic layer

17:48capability as it comes out. That's been teased for about a couple years now uh in Databricks. But yeah, like I said before, the uh not just the workflows, but Databricks just having a unique capability of having entitlements where you can grant people domain access like if they

18:12uh have more of a data warehousing persona or a data engineering and machine learning persona that you can like give them the type of access within the platform that they need. I think is a very awesome capability that I hope continues to get built out to even other roles like if somebody does metadata

18:34management inside of Unity catalog. uh but I would say yeah a lot of the data engineering and uh data warehousing capabilities are the things that I have the most affinity to and and what the announcement that you may I don't know you you would expect from the uh from the data and I summit like maybe I don't if you just try to guess I don't know

18:57it's every year it's harder and harder but maybe what what would you expect I assume there's probably going to be another crazy acquisition uh there there might be some another acquisition like there was recently just like even Bladebridge but I I am really fascinated to hear more about the AI

19:21data masking capabilities that Databricks bought a for. I feel like that one has been cooking for some time and I'm hoping to see that one pretty soon. And I know we have like the ability to automatically just or to just manually go specify row and column masks, but I'm really looking forward to the platform

19:43using uh context to be able to go help you do that in a more accelerated pace. Yeah, I mean you you raised the bar. Let's see in the from the 9 to 9 to 12 June, what would be the key announcement? maybe a new acquisition as you mentioned or maybe more details about maybe the integration of Okera

20:05it's more on the uh data data data discovery and also uh data masking and I think a lot more a lot of things related to uh like Apache uh Apache Spark uh Delta because last year they announced like Delta Lake 4.0 Spark 4.0 know and they also talked about the

20:27integr the acquisition of tabular so maybe we can expect some integration with the Apache iceberg as well so let's let's see I think there are a lot of exciting announcements I mean even even internal folks working at Databricks are not aware of those uh those announcement so so we're also all even

20:49the employees are expecting those um those announcements and anything else to Andrew, uh, before we close, I don't know, maybe some feedback, maybe some advice, something maybe you have on top of your mind. Um, I think, uh, that it it has really been interesting to work with data

21:11bricks with how open they are to product feedback. Uh so you know new capabilities or competitor capabilities come out and if you're able to articulate well how you think uh the platform should continue to evolve that's always been basically invited with open arms or even even when

21:34you challenge uh things how they they work a little bit. Uh so I've been really intrigued by the continued even commitment to openness that the platform has uh and realize like not 100% of the things can be open but even just the

21:56courageousness to open source something like Unity Catalog live is just uh pretty awesome to see that you know there's this whole other side of integration where somebody could even use those technologies and stand up their own lakehouse type platform with their own engine. And I'm super excited

22:18to hear about uh organizations that start to do things like that or start to even use other capabilities like uh credential vending and now there's all these conversations about interoperability. So I think there's just never a dull moment and having being able to work with Databricks a

22:40lot and not and be able to just articulate changes that that could potentially help catapult the P platform forward has always been really nice to and I haven't taken that for granted by any means. Amazing. Thank you. Thank you so much uh Andrew for your time. It was an amazing talk. I'm looking forward

23:04to meeting you at the data summit. Yeah, thank you. It's a pleasure to be able to talk on the show here.