Liquid Clustering & Delta Lake Optimization: Author's Best Practices

0:04Welcome back to the MVP series. Today I'm with Scott and I have for the first time Quinton as a co-host for the MVP series. Hey. Hey everybody. Hey Scott. Hello Quinton. Hi Yousef. I'm super super happy to have you on the on this talk and and and I think I should congratulate you for your awesome uh MVP

0:26swag. I really love it. Thank you so much. It just showed up in the mail, so I'm very excited to be wearing it this morning. And can you like introduce yourself to the community? Yes. Yeah. So, I'm Scott Haynes. Uh I've been working in essentially like data like data intensive applications for about 15 years now. Um started off um like a front-end engineer. Uh got really

0:49interested in like analytics and trying to figure out why applications weren't working the way that they should work. And I sort of kind of fell into analytics uh by accident. Uh it wasn't never something that I really thought about. Wasn't something like they taught. I was in school in like 200 like uh 2002 to 2007. Um and it wasn't something people were really thinking about too much. Um and so I really got

1:11into like understanding how the world works through data and that's been now like the passion for the past like 15 years. So um accidentally fell into something I love which is great. And now I'm just kind of curious about uh continuing to find more and more interesting ways of using data to solve problems. Uh outside of that, um I also make hot sauce. So you might Wow. Okay.

1:33We need to talk about that. We can talk about gardening as well. Like it's all little outlets. And I heard you also write books. Is that correct? Yes. Yeah, that was um Yes, that book right there. So Delta Lake definitive guide. Uh that was the second book. Um that one was like a labor of love. Um, anytime you write something with many people and you kind of, you know, converge everyone's

1:56ideas into something that kind of reads end to end, it's a lot more of a struggle than if you're just writing the book like yourself. Although you have like a, you know, it's like a team or a book family, like a writing family to kind of bounce ideas off of. And also like if you know, you start to slow down on a chapter, you get motivation from everybody else who's continuing not to, which is really kind of fun. Um, plus it's just, you know, it's better to

2:19always have an idea, bounce it off of other people as well. Um, so it was everything takes a long time. Like people see books and they don't realize that it's like a year and a half of people's life that goes into it. Um, and then that that's only just getting like the final manuscript out and then there's like months of going back and forth. Um, but you also get like there's like this whole entire team of people involved in it. um like all like the

2:42tech reviewers and everybody else and in a lot of cases like you meet like these kind of long-term you know friends right it's interesting like the O'Reilly community is a lot the same as the Databricks community um a lot of the MVPs like Audi Pollock was helping out with a lot of like the review and she's also just been somebody that I've had looked up to you know throughout my entire career as well um so it's nice to

3:04finally work with the people that you know you look up to and books help with that yeah it's great do you have some metrics on the books do you know how many how many copies have been sold so far. Let's just say like many copies. I don't know. It's like it's not not bestselling because we're giving it away, right? Um so like if you go to like delta.io and you scroll like halfway down like the homepage, it's free. And so like with that, every

3:26author going to the book knew it wasn't going to be a bestseller, but we still wrote it because everybody Well, I guess I was the only person outside of Databricks who was working on it. But I was really curious about going deep into Delta Lake, understanding the things that I didn't know. And there's like no better way to do that than if you're writing a book because you feel like an imposttor the entire time. You're like, "What is that thing that I don't know that I want to know that I really don't

3:49want to sound like an idiot if I get it wrong?" Like every time you write something, you're like, "Oh, that's probably true." You're like, "Wait a second, but is it?" And then you go into like two to three hours of like research to make sure it's correct. You go all the way back through the Scola code. you go back like now through the rust code as well making sure that you understand exactly what you're saying. Um and that's something that you usually don't you don't get that time on the job like so writing gives you a lot more of that

4:12opportunity to you know dive in deep. So yeah and I feel like it's always a good learn to like it's a good way to learn. Yes. Um if you have to explain it to someone especially in a book like you really have to learn you have no choice. Yeah. You also don't get to edit, right? Like if it's a blog post, you're like, "Oh crap, let's just make an edit. It's fine. Let's go back." For a book, it's much harder to make an edit, especially

4:34if it's already published and polished. And you you have to wait years. They have to wait for like a second edition uh to get those to get, you know, just making minor updates with that. So, and I remember I was also part of the team who like were was reviewing the book and uh I remember I told them hey I think we're now like now we have liquid clustering and I think you should really

4:56release like a chapter about this uh about this feature because it's super cool and then I I think Denny or someone told me okay we're like I think the books have been delayed for six months to make sure that everything is aligned and we have maybe the latest features that we can add because we're keeping more and more. Yeah. So Tristan got liquid clustering in. Um that was like

5:19one of that was fairly late into the game. Um it's interesting though cuz like we started the book in Delta 2 and then Delta 3 and then we're also priming for like Delta 4 right before everything came out. So it's interesting because it's like there's a lot of stuff that a lot of stuff changed along the way. Um so I spent a lot of time talking about like partition optimization. That

5:41chapter could have been a lot shorter because it would have been like let's just turn on liquid clustering and here's where to use it and here's where not to use it and then you know it's almost like a little sidebar where it's like and by the way it's like you can use traditional partitioning and optimize in like zorder by which would you know which would be great but you know hindsight's 2020 I can can you share your like your secrets uh and your tips on when to when to use it when to

6:04avoid it you said the order do you still rely on the order I feel like you shouldn't anymore no So I think I mean so if you think about like the difference between like bin packing right like I just want everything to be clustered as close as possible you know I want to fill these files up um I don't really want to have as much data skew um that's only really possible when you don't have like uh directory boundaries

6:27and so like with traditional like hive path style partitioning you're going to always have some small files depending on how you've optimized like the layout of the table and so we had a session earlier this um we were calling it like beyond clustering and the reason why it was like beyond clustering was just because it sounded really cool. Um but essentially it was more of like you know what's like that split point right where

6:50it's like you know I have a table and in a lot of like classic like data engineering problems you have a lot of siloed data that you're doing you know contiguous like partition overwrites and so if you have a lot of partition overwrites coming in overwriting large portions of the table then everything you did for incremental like prior optimize whether it's liquid clustering or bin packing like optimization those cost savings you know for read go away

7:13again and so you're like Okay, well let's just do that whole process over again. So you don't really get that like append only and optimize incremental type flow which is like so powerful. So instead it's like well you know do this not so performant thing for a limited amount of time like you know keep that retention window small almost like a close the books like for a lot of data

7:36and then after that move into liquid clustering for more of that historic like long-term memory and so like the advice has essentially been contextual. So in the case where you're continuously doing like you know upserts it can be difficult um you can still do it right like if you think about like right optimization like you can do it in process um or if you're running a workflow you can do it as a very last step of that workflow um but in a lot of

7:59cases it's like what are you optimizing for like are you optimizing for an immediate downstream customer or are you optimizing for an eventual future where people may read something and so like a lot of times like we like uh internally like at work like I work at Nike and we talk a lot more about like you know when is it needed right it's like just in time like can we do things just in time and save the money or do we need something to be fully optimized because

8:21it's like one of those mission critical use cases that everyone needs this data right now as fast as possible and so like you'll you'll make different decisions depending on you know what's the SLO right and you know how fresh does the data need to be so so if you only have like append kind of simple I would say yes so you're saying if you start having a lot of upsert and updates that's where it goes tricky key like

8:43remind me how like can you do like can we have one flow doing the obser let's say know the app and the updates really quickly and then if you run an optimized in the back end could you have a conflict like I guess because my thought was can you do like I know every night just you know once every night you just run the uh the clustering option but could you have a conflict

9:06doing that so not if like it depends on it depends on how like how the table's laid Um it like if you are continu if you are continuing to write to date partitions for example and then you're clustering within date partitions you can do something like that. uh if there's a chance of what you're optimizing to conflict because you have

9:30like this variable window that you're writing to, then you can't guarantee that a right is going to hit a brand new, you know, table, right? Not not a brand new table, but like a brand new file within like the actual table itself depending on that strategy if there's an upsert involved. And so like that's the only thing. So it's like if you have like um we talked about like Kappa like the Kappa architecture and it's like you

9:52can use some of the same methodologies where it's like you're writing quickly um you're not really optimizing that as much like you could be taking you know essentially kind of like snapshots as like a virtual table. So here's my view of that table. Um and now we can cache this view off for a little bit of time. Um, and there's like there's like kind of one trick to being able to, you know,

10:14kind of pretend things were laid out well because now it's fast because it's cached. Um, versus actually having it be, you know, truly laid out correctly um, and clustered optimally. Um, but in a lot of those cases like you're really looking for like read and write op like isolation and if you can't guarantee that isolation, then some of these processes become a little bit more tricky and you'll get like a concurrent

10:37modification exception. And if that happens, it's okay, right? Just, you know, it's like regular check-in set. You back off, you try again. Um, but you know, or you just ignore it and you just pick it up the next time. Um, because eventually you'll be able to make those optimizations. But it's I think it's difficult too cuz like like I come from like kind of like the streaming world and like streaming

10:59analytics and like streaming click stream type stuff, you know, that's easy, right? Because like I've done something, right? I I did some action. Now I'm emitting that. Awesome. I'm going to do other things. It's all part of like a story and I'm going to collect all of that like as you know a sequence of events and then I'm going to summarize that into something and you know that's going to encapsulate some sort of behavior. And while it's more

11:21complicated because there's more things to keep in mind, it's actually a much easier process than a lot of these like, you know, multi-siloed, you know, geodistributed type problems where it's like we want a snapshot of what happened everywhere, but we have all these different systems writing to this table and then you run into like more of like that multiight, you know, multi-write problems are always, you know, the biggest issue. In a perfect world, you'd

11:43have, you know, it's like the game of telephone and at the very end somebody's writing on your behalf. Reality is different. I remember we met uh last year at the OSS summit. Yes. During the Rust session like Rust track and I want to know like your opinion because like I start seeing more and more folks like coding in Rust and same thing like for Delta community

12:05like more and more people and customers are using some features from Rust for to use with Delta. So what are your thoughts about this? Um I I think it's a great I mean it's a great option. uh we have a session eventually coming up um like eventually like within the next two months where it's essentially like the do's and don'ts of like using delta rust and a lot of that is it's because you're

12:27not clustering these operations because you're fitting this you know if you think about kind of classical like horizontal or vertical scaling a lot of like the rust optim like what you're doing in rust is a more of a vertical scale I'm going to be processing this on a machine versus in a cluster for now right that's just the state of like the delta rust So there's certain things where you would run out of the ability to process

12:49on a single machine and at that point you want to go back to like kind of the classical like you know sparkdriven delta but a lot of use cases aren't big data like unless you're working at a gigantic enterprise a lot of cases in most companies are essentially like shorter it's like shorter bursts of information that need to be you know processed as they come through and so

13:11that's where like whenever like you hear like uh Tyler talking about like the, you know, the heyday, like the early days of like the CFKA Delta Injust project and now like Project Oxbow, like a lot of that was we have data in CFKA and all we really want to do is literally distribute that across partitions to a bunch of different like uh Delta Rust like receivers and then they're just going to feed it, you know,

13:34into Delta Lake table. It's like cool, awesome, we got that. Um if you want to then do a lot more kind of complicated like transformations then that depends on if that can scale on box like how long do you need to keep you know that memory around for like instead of just thinking about this as a continuous stream are you processing a lot of information do you need to shuffle and if you have to shuffle because we don't

13:56really have like that concept then you need to go back to like a different type of compute you know system um to be able to go do that and so it's like for a lot of it it's like you know a lot of data engineering problems are I'm taking I'm taking data from one system to another and for a lot of those use cases like that's an extremely simple way to do that or if you want to process a lot of things like using data fusion you can do all that in box as well and then it opens up different ways of working so if

14:19you think about like you know a lot of like the you know distributed systems or systems engineering problems you could do a lot of like transactional log you know um processing using delta rust on box and then you could send like results downstream for further processing as well and so it's just giving us different ways of kind of mixing you know those capabil abilities. The same thing if you think about like Delta Connect, right? You can connect from

14:41different systems outside of like the normal, you know, JVM side, you know, JVM architecture based, you know, processes and that opens up the doors to a lot of different processing as well. And so Delta Rust for that is also feeding into like what's starting to be kind of like you know I feel like it's more right now it used to be like all JVM based and it seems like all new projects are all rust based and so it's like you know it's a very kind of clean

15:04you know community member for every other rust project as well you know they all are working with the same you know it's all basically you know it's all arrow buffers and like data fusion for a lot of cases so that interoperability becomes a lot easier and for those who are listening and maybe doesn't know what's delta connect. I advise you to have a look at the release notes of the Delta 4.0. Delta Connect is part of it

15:26as well as the type widening and collations and the and much more. I I I always keep struggles to remember all the features we're adding in Delta 4. There there's a large list of them. Uh if you if Yeah, type widening is a big one. Um, I think for people that want to not also be in the JVM, you know,

15:48sitting in the JVM community for a lot longer. Um, there's a lot more kind of interesting things happening with Delta Connect as well. But that's a good kind of throw out to, you know, where we're going over the next couple of months anyways with Spark 4 and Delta 4. So, how did you end up uh being Databricks MVP? I mean, for my case, I think I I've been following you for a very long time

16:10in Twitter because I was I was a big fan of Delta before uh like since I think 2020. Yep. But like tell us the story. How did you become an MVP? Yeah. So, um I also I have not a not a real good like uh clue on how I guess, right? Um, so I was part of like uh the Beacons program before it was like the MVP pro

16:33uh program and that was like that was during 2020, you know, right? It was like I remember just like getting some stuff in the mail. It was like uh it was like a canvas painting of my face like welcome to the beacons um and like some stickers and all this other stuff and then like it was like this like uh JBL like uh like boom box type thing that had like the beacons be kind of painted on it. I'm like oh this is really cool.

16:56I'm like, I need this right now cuz like I've been sitting indoors forever and I'm like this is like nice to get like a gift from the outside world, right? It's like during those times and um I I I think like I found out later it was because of just like uh it was the work I was doing at Twilio like creating like the Spark community and helping to kind of drive that. Um and a lot of like the you know conference talks and like just writing online like a lot of the

17:19medium posts and you know different articles that were you know teaching people and I used to do like a lot of different like trainings and workshops and meetups and you know it's like here's how you make things easy um you know here's how you test your applications um this is what like a pipeline looks like this is a simple way of doing like config driven pipelines and you know things like that and so I have no idea who voted at all but you

17:41know I'm really happy that somebody did um you said. Yeah, thank you for that. I think he must be Denny. That's for sure. I think. But we should we should definitely we should definitely ask him. Yeah, it'd be it'd be interesting to find out. I mean, I've I've been friends with Denny since like 2017ish. Um right about then. And then um yeah, I've been also friends with

18:03like Jules because they used to run like the Bay Area Spark. So I had reached out to them about like just hosting some stuff and it's like hey like how do I get involved? like I want to you know extend this meetup at that time like when I was working at Twilio we were everybody was like on like El Camino and like Mountain View and so like down the street was Reddus Labs up the street was Databricks and like there's a lot more kind of it's

18:25like we could just walk up and down this corridor and you know basically just do meetups going from company to company and like right in the middle was also like elastic so it's like just hop up the street you know and I'm like well what if we did like uh Twilio just opened up like a rooftop deck and I'm like we could use it and then the pandemic hit and so a lot of that work happened and you know just trying to get like you know it's like all the signatures and everything else like you

18:47know all the site policy stuff that goes into like opening up the doors to an office and so it's like all right cool now we can do the first one which we never did but you know that's okay that's history that's behind us um but it did lead to like a kind of long-term friendships as well which is cool so I guess like advice for people is like if you're curious about something reach out to people who are running it right it's like you never know what will happen

19:09yeah and it's surprisingly more like easier than people think. Usually everybody's happy to talk to someone and and it's working well at the end. Oh yeah. I I mean with Slack it's like it's like cool everyone can talk to anybody. Um but I think it is interesting though because people usually like people assume that no one has the time for them which is what I found where it's like

19:31hey do you have like two minutes to answer a question? It's like of course like people I mean people have time or you don't and eventually you'll get back to it as well and but you know I think most people to your point Quinton are very much open to helping people. I think those are also you know it's good community aspects as well. I was wondering can you can you share a little bit on what you're working on at Nike? Maybe not on the you know specific of

19:53the business but more like the kind of I don't know pipelines you have. Um yeah what's going on here? Yeah. So I mean um if you think about like any kind of like retail e-commerce um we're working on everything from you know the analytics and like you know machine learning on like supply chain to understanding more about like you know

20:15it's like where product goes why right what are like the customer like what types of customers are you know can we you know can you group together right like anonymously um you know what are the things that I purchase what are the things that other people purchase that's a lot like you know Nike not like Nike recommendations, but it's like the Netflix recommendations engines, right? Where it's like if you think about like retail product as, you know, the same thing as a movie. There's different

20:37characteristics of each individual, you know, product, including like fit and everything else like that. And so, if you want to be able to go and do a recommendation for other people, it's good to understand whether or not people have enjoyed something and they like something and they've continued to purchase that. And if they do, then it's like it's a little bit easier to be like, hey, we recommend this other part, this other thing because we believe you're going to actually like enjoy it. Um, but a lot of those like those those

21:00problems aren't like the those aren't the hardest ones. So like a lot of like the problems that I do deal with are like you know how do you deal with like uh you know potential you know like potential issues in like a matrixed organization right like if you you know and I think those are the harder things too right it's like here's like a wild idea we have a couple people that are on board for this wild idea and now how do we get that you know essentially

21:23approved by leadership to move forward as well and I think those things are kind of fun um I say kind of fun like it's actually really really fun to you know spawn a new idea and then get people to work work on that as well. Um you know it's kind of like uh doing hack days as a career which is neat. Um but for a lot of like a lot of the work it's just you know it's how do we ensure that product continues to get delivered on

21:47time right um that it's what people want so that stuff doesn't sit on the shelves um things like that. So how do you work as a team? Do you have like many small squad and everybody's doing you know stuff on data or do you have one central squad and everybody's relying on them on maybe some kind of expert and you can pull the expert based on the project. Um so it really it really like it like it

22:09depends right. Um so you have like there's certain things where it's like you have like you have focus groups of people that are always working within like that same domain right if you think about consumer domain for example like commercial domain like there's specific things that you need to have kind of like you know it's like intimate knowledge of like the ways of working to be able to really excel like within like that you know subset of the company. Um there's other places where like we do

22:32have like a lot of like the principal engineers and senior principal engineers will like we talk about like the Mary Poppins effect right they'll go work with like a team for like you know two weeks to a month you know it's like you know swoop in help the team teach them a new idea um you know train them in things that they need to understand um if they don't currently know that or if like they're just kind of getting up to speed maybe it's you know new architectures etc and then they can like

22:55kind of pop out and go help other people as well and so it's kind of like a hybrid like tiger team um for some of that work um we keep on trying different things right where it's like you know there's always like more and more and more things to learn to know and then it's like you know education at like a very large company like for example like the org that I work in there's 1,800 people and so like with 18800 people

23:17like within like engineering it's like updating everybody to get you know not everyone is going to be completely up to date and so it's like what can we automate like what do we teach what do we train like what do we do as like internal like meetups things like that Um, and so there's like a, you know, a large effort that goes on, you know, inside that no one talks about. We can't go into details about a lot of it. Um, and you can't really record a lot of it

23:39either, but like we have like inter like entire like we have like a lot of like it's like on-site like internal conferences and other things like that as well where it's more, you know, knowledge sharing and, you know, teaching of like new frameworks, things like that as well. We also do some open source and, you know, I don't know, it's it's interesting. It's probably like a lot of other companies, right, where it's like it's like, yeah, we have teams that work on stuff and in some

24:01cases like they'll be working on the same project for, you know, for years and other cases, you know, they'll be working on platform components until those are stable and they'll move from there. Uh yeah, every day, every month kind of dictates new work anyways, right? And so go with the flow. And with your like I know you have like a very extensive um or large experience with

24:24the Nike, Twilio and other companies. So what can be your advice to aspiring software engineers or maybe data engineers like how can they be the best let's say keep keep learning right like I think that's like the biggest thing like that like for anybody right it's um it's like kind of like I think about this thing where it's like it's not necessarily like a mantra but it's like kind of never settle um like never

24:47settle for status quo never settle for just knowing what you know and like always be curious like there's like there's no experts right like the second you're like I'm an expert like you've probably already become you know mediocre again at what you are doing. So it's like like ne never stop learning that thing if that's like something that you're passionate about. It's like the same thing with any kind of language as well, right? It's like just because you

25:10know Python really well. It's like do you want to just do that forever does it make sense for you to you know go back to Java? Does it make sense for you to learn Rust? Does it make sense for you to learn Golang? What are the things that kind of keep you kind of nimble? You know, it's kind of like every time you're doing something different, it's like another kind of brain test. Like I think about like there's an old game for like Nintendo was called Brain Age and just a bunch of games where it's like,

25:32you know, it's like solving simple math problems as like things are kind of coming down from the sky or like you know, remembering a recipe and like like it's like kind of simple like you know, it's like uh people talk about kadas and things like that for karate. It's the same thing for engineering as well. Like is there something new that you've learned every week? Um, if you've learned something, you think it's exciting, can you teach it to somebody different? Um, you know, have can you

25:54write about it? Like is there something that you can do to like, you know, spread that kind of joy of something that you have learned as well and kind of spark curiosity for other people? Um, I feel like with that, like a lot of it is like, you know, just continuously doing that, right? There's always something new to learn. Um, and that can be applied anywhere almost. Are you using AI for these kind of learnings? I feel like now, you know, sometime I

26:17don't know how to do something, I just ask BI and like, "Oh, we can do that." Wow. Now I I have a lovehate relationship with like Copilot and like other other like uh immediate information retrieval. Um I I feel like it's like all right, I will use it after I've burnt myself out trying to find something. Um cuz I feel like it's I don't know. I like type

26:40ahead, right? And I'm like, okay, it's just type ahead, but it's type ahead that understands exactly what I want to do and it's predicting what I want to do in the future. Um, like the joke is always like it's like I'm going to go write this code and it's like tab tab tab tab tab tab. I'm done. Like that's good and bad. Like I feel like if I if I had and I don't know if this is like if this is going to come across correct or not or like politically correct or not

27:02but like when I first started off like early on in my career if I had something like you know co-pilot um I don't know if I would know as much now because I wouldn't have to and I don't know if that's a good or a bad thing right it's like you know if you need to understand like you know just kind of basic architecture of what a application should look like to be able to describe what the application design is to like

27:25an agent. Like if that's like where we're going in the future, there's going to be a lot less like it's going to be more of like you know it's like here's my application blackbox. Like the agent understands how it works, what it's supposed to do. I've given it a couple like you know parameters to work within and now I'm kind of hands off. Um I think I'm just kind of old and I'm like h I don't know if I want that. I like to know how things work and so it's like I

27:49don't know. I think it's really good for like you know automating you know automating PR. So there's a lot of different cases where you know if people are all busy and you want to have like a second opinion it's good for second opinions. Um and then in a lot of cases as well right it's very very good for understanding how to how to teach you how to optimize something that you might not have thought about as well. Um so if you're learning new languages etc good yeah it's super good for

28:13reviews. I turn it on on the demos that we have and when we do a commit it's like hey are you sure that's what you want to do and then you're sh like maybe some maybe not maybe I should go back and think about that again it but I think I think that part is good right um it's an unbiased conversation as well whether it's biased or not like you're not thinking about it as like a human and sometimes

28:35like sometimes people don't want to hear certain things like during the time of a pull request it's like hey this could have been optimized better. Um, you know, this isn't as testable as it could be. Um, and some of those conversations like they're good. Like you have to have them because that's how everybody learns, right? To, you know, write more performant code, write better testable

28:58code, more maintainable code. Um, but I feel like having a computer do that for you, um, is great because you can't really yell at Copilot and be like, "You don't know what I was trying to do." It's like, it's like, "Yeah, but I don't know. This is what I think you should do." like there's no like that human part goes away which means that it's more objective which is I think better for a lot of that um because I think subjectivity ends up you know being the

29:21heart of a lot of like you know uh elevated conflicts like within companies um yeah yeah that's right yeah at least you can get you know the feedback and that's you know just just the truth and that's it yes it's all we're searching for and I feel like it's a like you mentioned an interesting kind of paradox the I I feel like now the AI is going to

29:43be maybe in a few years like it's going to be able to do most of the I would say like the basic coverage uh engineering job most likely and then I'm still like I think we still have you know many years where the like the human is really going to excel on very specific advanced task task but I feel like to be able to learn that and become that kind of expert you have to do yourself all the

30:05average stuff and learn you know the basic first but now that it's done by the AI it's like removing you from the need to learn that So I wonder I wonder if we'll have a you know shortage of really kind of principal software solution engineer in the in the next few years because of that. I've actually read like I've read a couple different articles like on that. It's like more of like it's like that future facing like you know what if in the future type thing and a lot of it it's like it's

30:28like you know the conversation usually goes um you know you have somebody who's like a classic kind of like software architect, data architect etc. who is now filling in the parameters for the system. But usually that's people who are like, you know, like my hair is not looking super white today. But like it's it's usually people who've been like in the job prior to a lot of like, you know, a lot of like the AI and everything else like that, right? Like

30:51like we all had to suffer back in the day, right? You know, we didn't have debugging tools like we do nowadays. Like it's a lot of like those lessons learned, people will still have to learn them to your point, Quinton. Like if they don't learn that, then it's going to be a lot harder in the future. Like my generation ages out, right? and like we retire that's okay, right? That's natural part of the life cycle of everything. Um we still like like you still need to

31:13have like the knowledge experts in to be able to then like assist the systems. Um and then eventually you know the systems would be like we don't care you know we're going to do it our way and you like okay well you know just don't spend too much money right it's like all right um but I think for like the years to come like that's going to be a thing. Um I wonder more about like I don't really see like a lot of like you know it's like human rebellion coming from this

31:35but it's sort of like you know how do like how do we kind of change like you know people's perspective on like you know what's what's good and bad right like what's the right way to do certain things like I think about like um if I can like autoroot cause anal like autoroot analyze a problem because I have like a an agent that can look through logs awesome that's so great because I don't have to do any of that

31:57work I can't be everywhere at once like if you're say you're debugging things like in Kubernetes, there's a lot of different pods to look at. There's a lot of problems that could happen in like a large distributed system. being able to automate a lot of that like search and you know it's like kind of the old search and destroy mentality but in this case it's like search and analyze and you know then can I automatically you know reconcile that problem in a specific way whether it's you know an

32:20automatic you know pull release to a specific you know codebase to go fix the problem that can then go get released automatically if I approve it or if it just decides to approve itself like I feel like there's certain things that it's more of like that kind of classic toil work where it's like I don't want to do it I don't think anybody else really wants to do it either. It's like can we get rid of that work cuz like a lot of like if you think like not just

32:42like systems design or anything else but like pipelines, right? A lot of like the work in pipelines is like you know it's like oh something broke somewhere. Let's go find that and let's go optimize this pipeline to go and not have that problem happen again. And in certain cases, it's like, you know, if the same thing's happening over and over and over again and somebody has continuously, you know, fixing those problems andor just like, you know, recovering from those problems

33:04without fixing them correctly, it would be much better to have a, you know, a friendly assistant, you know, an agentic assistant come in and be like, "Hey, by the way, did you know this? Here's your new, you know, here's your new codebase. Um, we fixed it or we fixed this pipeline." Um, I think that's a lot that's a lot like I don't know, it feels safer, right? I don't I think I also just don't want to like give up control either, right? So, I think it's hard for

33:26me. So, but I think as we get, you know, as we kind of get, you know, further into like this, you know, the new world we live in, um, I think more people are just going to be accepting that, you know, this is part of how we do everything and, you know, how do we work better together, you know, with our robot overlords. Indeed. And and I think just before what you've said, I remember like just six years ago I had to go to

33:49Stack Overflow whenever I was coding in Spark and facing a problem and I would spend maybe one hour two hours in stock overflow looking for for someone who posted the same question understanding the response and go while now you can just copy paste the log but I think it's also taking an abstraction because now people doesn't analyze and sometimes they take what the LLM is responding for granted or maybe it's as the correct

34:12answer which may be wrong because it depends on the source used for this specific response. So it should always like um uh try to deep dive a bit before like take it for granted the response. Yeah. But I mean I mean it's always been the case like people have been on Stack Overflow just copy pasting stuff. Do you trust what you copy paste? Do you even

34:34understand? And now just you know giving you more of Stack Overflow out of the box. But I guess the same issue at the end. Yeah, I I agree with that. Like I think though like I think it's interesting right because I think it was uh maybe it was like two years ago or so a colleague of mine like a shook was going through this was prior to uh spark 35 uh prior to like having like

34:56protobuff like the from protobuff and to protobuff we're looking at creating like a uh protobuff 3 expression encoder and so we asked chat gpt it's like hey can you put together an expression encoder for protobuff 3 and it was like it took maybe an hour and a half of like fiddling to fix everything to make it work. Um, which I was really it's like this is awesome, but it was also giving

35:18us like it's like here's how you would use it. And like that stuff I think was really cool because like with a lot of like the kind of like you know deep in like the heart of like the Spark ecosystem a lot of times making complex changes are very very difficult given like the size of the codebase. The same thing could be said for like the you know the you know the Delta code base as well now cuz it's getting larger and

35:40larger as like the project continues to have new features like being able to like quickly get to where it is that you would need to look to go and solve something I think is also an interesting like aspect as well. So whether or not it's the stack overflow effect but using it as like a really you know a deep you know deep smart search engine for code is also really cool. Yeah. Yeah. search

36:02learning learning engine. Agree. Cool. Well, Scott, thank you so much. You're very welcome to have you. We're waiting for the next book. Maybe maybe a nice back. No, that one is already done. Maybe maybe a new one. We'll see. We'll see what what what comes in the future. Um but the idea comes back and forth many times. It's like is it worth the effort? Is it not worth the effort? Is it a good idea? Is it a bad idea? So,

36:24we'll see what happens in the future. So, all right. Thank you so much. Cool. See you. Bye, everyone. Bye.