Photon Engine & Incremental Innovation: Building Engineering Teams

0:00It's really important to be looking at whether what you're doing is growing you. Where do you want to be long term? We started using a lot of tools like cursor AI or copilot and oh my goodness, no one ever thought of doing this. The rise of genai. Hi everyone and welcome. Today we're excited to have Paul Aventis, VP of software engineering at Database. Hi

0:24Paul. Hi Ysef. Hi Quindon. Hey, nice to have you Paul. Yeah, we're so excited. And first, can you introduce maybe yourself to the community? Yeah, so I'm Paulus. I'm a VP of software engineering here at Databricks. Um, I joined Databricks about five and a half years ago, moving to Amsterdam from

0:47Toronto to take the role. Um, and prior to that, spent 20 years working in tech. Um, always for American companies from satellite locations. um some time at Google about five years and some time about 15 years in the semiconductor industry working on field programmable gators and the software for them. I was like looking at your resume in LinkedIn

1:09and it's super super extensive one but my question like how did you become a VP of software engineering and why Amsterdam because I know you used to live in Canada I guess. Yeah. Um so I it's fun story how did I end up at data bicks is straightforward in a sense and one day uh Reynolds one of our founders um had reached out he was just curious

1:31to learn about the Canadian tech market for some research he was doing into possibilities of what we do um I ended up you know he as Reynold by the way is a master recruiter he's one of our top recruiters in the company he's brought in amazing talent to this company but you know he is like obviously ends the call with hey so what are you doing and are you interested in doing something different and one thing leads to another

1:54and I was interviewing for a role different role with Databricks and uh during the interview um Ali heard that I was visiting a relative I have in the Netherlands my my mother was born in the Netherlands and my grandmother still lived here um and so I was actually that week happened to be flying out to visit her and he's like oh wait Netherlands h have you ever considered living abroad what about

2:16Amsterdam and I'm like oh yeah my wife and I were thinking about it our kids were eight and 10 at the time or like it's there's ever a time for us to go live abroad for a bit, this is a good time. Um, one thing led to another and you know, it was my, as I like to say, it was the first sort of data Databricks monkey wrench, as I like to say, where we often are willing to make very fast rapid changes and very biased for action and that was me seeing it in action from the leadership in that

2:39moment where it pivoted into a discussion of a leadership role in Amsterdam. So as a company we had a growing Amsterdam office and you know someone who done the Peter my my predecessor who done a great job growing the office to the size it was but we needed you know someone with bit more experience running a larger site to come in and help the site continue to grow. So that's how I ended up here. And then

3:02you know why Amsterdam besides the family connection? I've been here before. It's an amazing city. Um you know great place to live. I could imagine bringing my kids here and them having a wonderful time both in the city but also exploring you know Europe as a whole. Um so I thought it would be a really neat experience for for them and for my family as a whole. And I can assure the viewers this podcast is not

3:26sponsored by Amsterdam. Yeah. What's your uh which city do you like the best, Toronto or or Amsterdam? And and wait, be careful because I live in Toronto. I even put my Ben t-shirt. So, you have to Yeah. So, I've I grew up in Toronto. I was raised there. Um so, obviously I love a lot of things about Toronto as a city and my friends and

3:49family are there a long time, but honestly, I've come to really enjoy living in a smaller city than Toronto. So the fact Amsterdam's kind of a right-sized city um so I can get anywhere by bicycle safely without fearing for my life um and can do it in 15 20 minutes to anywhere in the city but also can be in countryside in 5 minutes and go for lovely long bike rides. So I I I secretly but don't tell

4:13anyone really kind of prefer Amsterdam if I had to pick. And and to go back to your like role at Databricks, can you tell us more about uh the VP role and is it mainly about managing uh or about team management like are you technically involved in the um in the architecture decision like or

4:37maybe do you rely on your team's expert to take those kind of uh choices? So at Databricks we have first we you know have highly technically engaged management. So you it's a company that's run from the top down you know in terms of hey founders in early VP roles of the company um who are very deeply engaged technically but all of the people we've hired after that we aim for everyone

5:00down to line managers up through the VPs to have the ability to engage technically with their teams. Now that said that's not always where our skills are most needed. When you have a team that has great technical expertise and are making awesome technical decisions, but needs to grow and you need to recruit, maybe you're spending most of your energy recruiting. Or, you know, the org is struggling and needs help

5:24with how it's organized, maybe you're spending your energy there. So, in general, as a manager, you're kind of balancing these sort of five different ingredients. You know, your strategy and your execution and the technology and the product direction and the people. um and kind of balancing these different factors and gap filling wherever it's most needed in the team. That's true at the lowest levels of management. It's true all the way up at the VP level, I'd

5:46say. Um but I've in my past I've at a Databricks I've managed directly a team working on the Photon engine which is a very technically complex um project. I helped execute the Databricks SQL product and get it out the door. that was more of an execution um and program management role more than anything though there were technical choices there and at other times I've been

6:08focused on how do you grow grow an office deal with um issues that you might have in a site so even I just walked out of a meeting with our works council so how do we deal with employee relations um how do we deal with changes to perks or benefits or the work environment or um last week you know there was a complaint about us serving only one meatball at lunch per person you know so the nature of my job is

6:30highly varied depending depending on the dayto day. Do you want to go? I I got you with the meatballs, didn't I? I wasn't expecting like a vit facing those kind of issues. Well, I mean, and again, like anything, you hope to surround yourself with a team of um of people who are awesome at

6:53their jobs in all the different jobs that we have done. You have great facilities people and great people on the, you know, HR and the legal and so forth sides to deal with many of these different issues, but you're helping to ensure that things that aren't getting dealt with or are escalated do get rerouted correctly and helping to debug and diagnose and problem solve, but in the end you're trying to match the right people to the right.

7:16So you you you just mentioned Photon and you was, you know, working on Futon with your team. Um I think it's a good topic because I wanted to talk about innovation a little bit on on on data engineering. Well maybe we can start can you can you share a bit more on photon what is it doing kind of how is it working at a high level and then uh I guess the followup question would be on on innovation like how do you drive

7:38innovation as engineer and I think photon is a is a good example of that. Yeah. So, Photon was um our effort in about 2019 2020 to 202122 and still ongoing to basically rewrite the guts of the Spark distributed data processing engine which is at the heart of how data

8:02bricks does um you know large data uh processing rewrite the core of the processing engine um in C++ and so it's writing it in a native language instead of Scala or Java but writing in C++ also um leveraging um simultaneous um sorry SIMD instructions in the processor. So basically how do you process multiple pieces of data with

8:25a single instruction um so this is a capability processors have but it requires relaying out data looking at data in a column there rather than row-wise fashion and handling things in batches. It's a very interesting approach that if you handle your data in just the right ways in the right orientation, you can you can leverage the layout of data in memory and the way

8:47processors want to process that data to do um processing in substantially faster ways. So you know if you think of a part Apache start spark is a inter is really an API an interface a way of doing data processing and then there's a open source implementation of all aspects of spark. Um but this was replacing at the lowest operator level

9:09how those operators are executed with Databricks proprietary implementation. So that's what the photon engine was and is was basically slowly over time handling more and more and more of those operators and speeding each of them up. Um in that case our target was starting with getting it to be 2x faster um for every single operation that we touched. Um and then how you sort of approach. So

9:32that's just an example of a great innovation challenge. That was one that was started by you know Reynolds saying hey we need our engine to be faster. Let's step back and build something. it's going to take us two years to get there and some amazing people who had incredible experience on this and amazing engineers working on this with who had various ideas and then just you know the way that the funny thing is a lot of innovation occasionally

9:54innovation is like eureka moments and oh my goodness like no one ever thought of doing this but sometimes it's just a lot of elbow grease it's like hey how do we systematically look at each operation um and you know make a change measure it make sure we've got the improvement we wanted and Actually, a hard part of it is deciding to move on and move on to the next thing. Not sitting there and playing with the thing that's fun to

10:16play with and get another point and another 10% when you have another target available to you that you need to make 2x bigger. That's a one thing that happened during that time. But the way that the innovation works there is, you know, there's a mixture of sources of ideas and engineers read an interesting paper or looked at the patterns that they saw in the customer data and said, "Hey, wait, we're kind of the way we're

10:38approaching the data is this way. If we oriented it that way, it would be faster. Or they're profiling it. They're brainstorming. We have outside experts who've seen things or we go to a conference and listen and hear things. And the key thing is that you're getting exposed to many different ideas from many different sources and giving and your team has the freedom to um think about those pieces of information and

11:01synthesize them and collide with each other and come up with you know something new. Um but a lot of occasionally innovation is this big leap but often it's just selecting the right problem to work on and setting a ambitious goal but you know then making incremental progress towards it. So if I take Photon it would have failed if we say

11:24we're going to rewrite the entire engine and replace the whole engine at once because it's extremely difficult to catch up with all of the capabilities and prove that you have an engine that is compatible. Instead, the key idea that allowed us to get there was saying we're going to take one operation at a time, the bottommost part of the tree basically, and say that operation we're going to make faster, and we're going to otherwise make the engine completely

11:47compatible, and we're going to peacewise peacemeal replace little parts of it until one day all of it has been replaced. Um and um that sort of approach was critical to being able to execute on a big audacious project like that, especially with a relatively small team like we had at Databricks when you look back five years ago. Yeah. Yeah. And yeah, I think what's impressive is

12:09when when you do that, it's like how do you know at the beginning that it's going to work at the end because okay, you start by the small piece. Uh but like I can really see like with such a project how you I don't know you start with something and then you realize maybe one or two year after or no you know it's not that good. See then I've seen that for example um I used to work at data stacks and they did something kind of the same idea they they wanted to improve the like the engine of the

12:32database they were working on and it took them maybe two three years to release something and they had to like they rolled back everything after 3 years like they basically destroyed everything. it wasn't working like they want it and they removed it. So I can see how very easily you can you know get trapped like that in in things like that. Yeah. I don't do you have any secret source to avoid that? Yeah. Well, I mean that's where wherever you

12:55can make an incremental change. It's always especially a production industrial kind of product. It's we don't have the luxury that you have in let's say academia where you can say I'm just going to start again and I'm going to show that this works fine on TPCDS or some benchmark and show that it's you know that it works good on this one benchmark. that's my only objective. I publish my thesis and I move on with my life. Right? It's um you have customers

13:17using the product. You have real real lots of real world cases that need to continue to work. Um so doing a complete wholesale replacement of a product um it's very difficult to ever get there. um when you're doing it in place um in place upgrades and peacewise um upgrading and changing things you can take advantage of techniques like AB testing where you can you can have both

13:39code paths available and you're doing a readonly version of the change under the new code path and double-checking that the outcome is the same as the old code path which is still being used in production or vice versa like you know these types of techniques anytime you can find a way to you know gradually change and be able especially if you can test in production without affecting

14:01your customers workloads or you have a way to very quickly detect regression and roll back in the case. But again, like if you release a big bang piece of software and then you have to roll it back. Your option is go back to the version you had 3 years ago, you're kind of doomed. You're never going to ever get to the to the conclusion. You have to be able to make incremental progress and roll back incrementally. Otherwise, you'll never get there. Yeah.

14:23And just I think just a reminder first for those who are like watching this recording. Photon is our vectorzed engine that as Paul mentioned like has been written in C++ and the goal is that it speed the process. You don't need to change your spark code. Basically if the function can be photonized then it will run on photon else it will switch fall back to the uh like regular uh spark

14:48engine. Exactly. And it's making those decisions part like not all or nothing. making the position the decisions on an operation by operation basis. So I think I think that's what almost everybody does now like Doug DB is kind of doing the same uh all the big players are doing the same. Do do you see a new like what's going to be the step after that? Do you think we will go to GPUs

15:10and do you have any any good um not really my area of expertise to be honest I don't know what the right next thing is um I mean my own personal background is in optimization software so um I've seen some of these trends over the years you know single threaded to parallel and SIMD and you know data layout it's always shocking how something looks like it's a solved problem in computer

15:32science let's take databases they've been worked on for what 50 years or more um and yet you keep coming across things that are new ideas that radically rethink how to do something or sometimes it's just an old idea that now because of the tradeoff of time that you of time or cost in the system like you know SSDs come along and suddenly it changes what does it mean to spill the disc it's very different than if you have a spinning

15:54disc or you know memory becomes cheaper or faster or these other things come along that change fundamentally the trade-offs in the system and suddenly an old approach that was worse is suddenly better or a new approach that no one's considered is suddenly better and you know I think I saw a paper I don't know five years ago maybe it was a little more now where someone had just you know beaten the record for in-memory string

16:16sorting you know the most kind of basicish thing and by just challenging an assumption of like pointer swizzling like saying hey I'm going to move the actual data around you're actually better doing this than you are um you know swi moving around pointers to data in memory in certain cases and so there's lots of these very interesting things where very fundamental building blocks then result in changes in the

16:39overall algorithms and approach. Um obviously changes in compute substrate. So moving to you massively multiparallel processors such as a GPU um you know can they could lead to massively different um performance but only for certain subsets of problems or only if you recast the whole problem a certain way. So I don't know. I don't know what the next one looks like to be honest.

17:03And I'm I'm curious to know what are the features that are currently developed in the uh in Amsterdam. I know for sure workflows, but what are the other features? Yeah, I mean we work on many different pieces of Databricks because we're we're the oldest satellite office of R&D in in Databricks and the largest one outside the US. Um so we have quite a few

17:27different project areas but our focus areas first is you know lake flow so that's work so workflows or jobs um you know part of the stack top to bottom everything from the UIUX through the control plane how we manage the data plane that's done in Amsterdam um but also we working now on the DT products so how we make um uh you know declarative um pipelines work in addition to imperative ETL like we do

17:50with jobs um so we're working on that we work most layers of the storage stack of Databricks. So Databricks is giving you the performance you know giving you a performant data warehouse and a performant um uh you know data lake on top of your blob storage from your cloud vendor. So GCP or your S3 from from AWS

18:12for example you've got your data sitting there and we need to give in incredible performance on top of that you know eventually consistent somewhat you know slow storage. Um so it's a very interesting problem to solve and so everything from low-level um how do you read and write as rapidly as possible in parallel as possible to the storage layers to how do you govern that data

18:34that whole stack? How do you encrypt the data? How do you um give abstractions of that of that storage system? How do you make three different clouds work in a way that's consistent um for our customers and for ourselves because we have a product that is agnostic of the three clouds for the most part um for our users. Um so basically that whole that whole surface is worked on in

18:57Amsterdam. It's another major area of focus of the site. Um we also have people working on all sorts of other problems on I don't know I'm trying to think of a few things like looking at Unity Catalog also falls into there. There's, you know, we're trying to make sure you can govern and secure those storage assets in Unity catalog and expose them as, for example, volumes. We have the all efforts on how you automate

19:20the use of Databricks through SDKs, a command line interface, IDE integration, Terraform, all those types of um capabilities. Those are all built here. Um you know we also have team smaller teams that work on int internal developer tooling how our own developers write code productively how life cycle of VMs that are used for compute are handled um and you know many other sort

19:44of areas um of the stack. So a lot of it uh at this point and I'm I'm sure a lot of folks would love to know a bit more about like uh the internal stack at Databricks. Do you do you think you could share a bit more maybe on workflows uh what we have how is it working at a high level? Yeah. So if you think of a product like

20:06workflows so workflows is a product that for those who don't know allows you to orchestrate your your ETL um on Databricks. So I have an you know a a sequence of tasks I want to operate on a trigger for example a time trigger. So, Friday at 9:00 p.m., run the following sequence of things in order. Run this notebook and then do this little blob of

20:28API calls and so forth and write the results over here. Um, you want to get this done. Um, there's a few different pieces of that software stack. At the top level, you've got a web app sitting in front of you. You're sitting in front of a web application. You're using a UI um for the form and for boxes and layout and so forth. Um, that's your classic web sort of stack. We're using

20:50um you know we're uh sorry just finding my notes here. Yeah. So um uh sorry let me uh let's cut here. Yeah, we'll cut here for uh I am I had the wrong version of my notes up so I'm trying to find it and it's got the wrong wrong data in it.

21:13Right. Okay. So, so if we look at the web the sort of the first layer, let's uh restart from here. Um, so we have the, you know, workflows, you have a UI where you're authoring your your workflow in it. Um, that authoring experience is a classic web application. It's written in React and TypeScript. Um, you know, communicating via GraphQL

21:37with um the APIs uh in the back end. There's microservices sitting in the control plane of databicks. So these are, you know, machines running running within uh Databricks's environment um that are handling all of the API requests and all of the back end that supports this web application. You've got um you know those are written primarily in Scala these days though um

22:00we're slowly starting to write some of them in Rust or a few things in Golang. Um but uh primarily we're a Scala house when it comes to um the prim uh the uh most of our control plane software and microservices. Um you know those services are reading and writing their state into uh you know my SQL and TIDB as um the vacuum stores. Um and then

22:25that's kind of sounds not much different than most full stack sort of development environments. What's kind of interesting and special is we're, you know, also as part of that workflow, we're spinning up compute on behalf of the customer. So we're acquiring VMs from a cloud vendor. So let's say in it's a it's a AWS, you're getting an EC2, you're getting VMs from EC2 and you're administering

22:48them on behalf of the user in the user's account um and executing on data that's sitting in the user's account in S3. you're installing a Spark runtime on that VM which then does the distributed data processing of the data on behalf of the customer. Um so all that life cycle is being managed under the hood in this workflows product. Um the software that's installed on the VM is written is

23:12things like that photon engine I mentioned earlier but uh Spark itself is written primarily in Scala. Um with pieces of it having been rewritten in C++ as well as Rust um in uh in places now. Um yeah so that's kind of the tech stack that we work on. So the majority of our work is done in Scola which is a JVM based uh language and that's for historical reasons because

23:35the original um Spark or Apache Spark software was written in Scala and that's where we evolved out of um but we are starting to use other languages where they make um make sense for our stack. I didn't know we were using Rust. I I know like they using Rust for some delta pieces but it's cool to know that we have Rust. That's where like you know you C++ and

23:58Rust are basically used in places where we absolutely need you know the maximum performance and we also want very good control over the memory consumption of the software um because uh you know a other languages can be more bloated and um that bloat can be problematic when it comes to efficient computation in the in the bottom of a stack. Um so that's

24:22that's the primary place where we're starting to deploy it here and there. Since you're talking about boosting performance, what about boosting performance of the software software engineers and I know with the rise of genai like we started using a lot of I don't know tools like cursor AI or copilot is it something that is currently encouraged to boost the

24:45productivity or yeah definitely you know our our our engineers are definitely looking at and encouraged to use um genis in day-to-day work in various ways. Obviously, you know, it's amazing for certain coding tasks. Um so where we have validated and approved the use of

25:07um these AI tools we um use them uh in our day-to-day tasks programming um and people keep discovering new ways that it's awesome and other ways that it's not so awesome you know and this keeps changing every every few months as these tools get better and better and what they're and the context that they understand and so forth. Um but you know examples writing unit tests is one of

25:30the things that um that the AI coding assistants are actually quite good at. They're very good at writing tests that test the code you've written. And in the process of doing that may you be like hey that's the wrong unit test. It's actually because your code maybe was not actually uh implementing the right thing. Um so that's an example of somewhere where some engineers have found that useful. Um it's very good for code fragments. Sometimes great for like

25:53large changes across large parts of the codebase. different people have had different success with different recipes. We try to share some of those um examples with each other um so people are learning from one another on how to be more productive with the with uh applying AI in their day day-to-day work. Um the other area is obviously non noncoding work. um you know it's like everyone these uh finding

26:16these useful for various forms of technical writing um or nontechnical writing even I don't know taking 25 peer reviews for someone and summarizing those peer reviews into a summary of the pros and the cons from the peer review or the areas of feedback um was a useful technique I used in the most recent um uh performance review cycle we did where I just wanted to make sure that my own

26:39summary of what I thought I had read from the peer feedback. Is that actually reflecting was I unsubconsciously biased here and I was I was removing information that um that was in fact present and I was just not listening to it. So it was a very interesting way to cross-check what my summary was with what a machine saw. Um yeah, that's a good point. You I I use

27:03it a lot also on on PR just you know when I send a PR I have my little agent and they come and they kind of double check everything for me. I think it's super useful. Um, especially for open source project. It's like you have your top senior advisor just checking everything you do behind you. It's super nice and it's unbiased like you know as you said like he's just going to you know the is just going to give you honest feedback and that's it. So it's

27:25really cool. Do you see that being used? Oh sorry go ahead. No sorry go ahead. Do you see that being used also on like in the learning steps? Um there is this kind of big discussion on uh what do we do with the you know kind of new data software engineer now because basically you hire someone fresh from the college they will have kind of the same level as

27:47an AI uh maybe not at their books because we do maybe you know specific engineering stuff but in you know a lot of companies that's kind of going to be the case um and then the question is okay so if they are just as efficient should I just get the instead but then if you do that how do you ramp up to the next level so that you can, you know, can get better than the AI. Yeah, I

28:09think that if your if your viewpoint is that you're hiring junior engineers because you need hands on a keyboard, then I can see why one might come to that conclusion. But I don't think that's the reason we hire junior engineers. We hire them because well, a they, you know, they are strong students. they have learned a lot of good theory and have learning how to design well and have critical reasoning

28:32they bring to the table that um that still isn't you know sometimes the AI will will only design the thing you tell it to or write the thing you tell it to. It's not going to actually figure out where the gaps are between what you know you know the problem in writing software often isn't how do I literally write the code that does this. It's what's the right thing to write in the first place. Um what's the right problem

28:54to solve? um you know, is this the right thing to build at all? And that's never that's not a thing that I shouldn't say never because these AIs are super super uh you know, impressive of how good they're getting, but like um that's that's not the you know that's a thing that you need really strong engineers um behind the keyboard to continue to do. Um so, you know, for junior

29:16engineers, AI's can be a great productivity boost. I mean, I think of it a little bit like, I don't know, when word processors came out and then, you know, spell checkers and grammar checkers and and now even LLMs. LLMs can write an essay for you. But, um, if you just hand in LLM written essays all the time, you never learn how to the critical reasoning you need to understand how you make an argument. Um,

29:38you can't judge whether one argument is better than another. There's a lot of, you know, stunting in your own development that arises from that. And so I think that the act of uh you know being being very good at reading code quickly becomes more important. I think it's like you know the AI can produce lots of code. You have to be able to read and look at it and learn from it, find problems from it, figure out how to adjust it to get to the real best

30:01solution. Um and so that that sort of insight you get from looking at what the AI wrote and then modifying it. Um I think it can be a great accelerant to your ability to write code. Um you know I think it also bridges language gaps. So for example, you know, you might know one language and not another. Um we're, you know, we hire people not because of a particular language they learn because we expect them to learn the language over time. Um but um with AI helping

30:25you, you can very very quickly start writing code in an unfamiliar language or to an unfamiliar API surface. um and gets rid of a lot of the whole hey how does this API on that service work kind of uh grunt work that you do as an engineer and lets you spend more of your time doing what you intended to do when you sat down at the keyboard in the first place which is build the intended functionality you're aiming for yeah

30:47know you're right and and I think like it's super it's super useful to learn new uh new things like new functions of the language or new ways of doing things sometime like I do that and then you know I would get like a new I don't know library or new function I'm like wow we can actually do that like super cool and then you learn about it and you can go to the next stage. And since Paul you mentioned recruitment, what are what are the

31:10skills you're looking for when it comes to a software engineer? Um, you know, I think we're looking for really strong problem solving ability. You know, it's your your the skill we want is the ability to learn, the ability to look at new problems and synthesize information, boil things down to um first principles like can you take a problem and figure out what is at the

31:32actual core of this problem? What's the problem behind the problem or what's the question behind the question, right? And that level sort of that kind of reasoning is what we're looking for. Um so obviously the ability to demonstrate this through you know problem solving live being able to write code solve you know write algorithms solve a systems problem these types of things like many tech companies look at but you're looking for that sort of ability to also

31:54adjust your thinking with new pieces of information or a hint and you software engineering is not a a lonely um you know u vocation. It's not one where you sit and you sit in a corner writing code. you are often working collaboratively with others to solve problems. Um so when working collaboratively with an interviewer, for example, can you take a

32:16new piece of information and turn it into um an adjustment to the solution that you're building because that's what the real world looks like um much more so than a than a statically defined problem. Um we're also kind of looking for um you know some signs that the person has sort of hustled or worked hard at something and that they're have some resilience. you know, have they what

32:38will happen when you inevitably face challenges in your job? When you when the job gets hard or when circumstances get difficult, you know, how do you respond to those challenges? And this is where we look at the nature of the experiences people have had and what they've done um in the face of those experiences. Um so those are some of the things we look for um you know along with probably a lot of the same things most tech companies look for people,

33:00strong academics. And for those who said we're going to put the link of the uh all the roles open in Amsterdam in case you want to apply, right? And and just to be clear, there's you'll see a list of roles available at data bicks and there's probably other roles that aren't even listed that you might uh be appropriate for. So feel free to apply anyway. And are you working remotely? is it

33:23you know mostly remote or are you going to the office? No, we're we're in the office um currently two days a week and uh going to be increasing that to three days in the I imagine in the near future. So um we do like to spend time with one another uh to increase the amount of collaboration. It helps also with the innovation that we talked about earlier. Yeah. And then plus because

33:48it's in Amsterdam so you can go super easily to the office. You can just take your bike and you're good to go. So like it's easy and nice and and regarding the um the uh the performance. So how do you measure the performance of an engineer? I know it's a tricky question but how would you measure it? I mean obviously there's a bunch it is somewhat subjective in the

34:12end but it's based you're looking at the impact people have. So the things they built you know were they did they work the way intend they were intended to? Did you you know if you've designed a software system, did that software system um survive the fires of launch and you know use in the wild? Um did it or did it fall over once it was faced with real real workloads? Or if you were

34:35designing you know a product was that product did it meet the job to be done of the user? Did it was it did it find product market fit? Um if you're a more junior engineer you needed to write code. Did you write the code cleanly? Did it work? you know, does it look good when it's reviewed? Do you work well with your peers on in iterating and improving the quality of that code? So, obviously, there's a number of sort of

34:57um outcome related um things we look at. You know, we so you're not measuring the inputs as much as you are the outputs in terms of what happened, not the for example, the effort that went in. You're also looking at whether you're making the world around you better. So, are you um helping your teammates be more productive or more um you know, are helping them improve? Are you mentoring?

35:19Are you um helping hire? Are you helping the code base look better? Are you building tooling that make the people around you more productive? These are all sort of leadership and citizenship types of contributions that we look at. Um and we're looking at also how your your basic skills and tools are developing. like are you starting to develop the skill set you'll need in order to be successful not only at your current level but at the next level. So,

35:42you know, the nature of software engineering jobs changes as you work your way up through seniority. Um the nature of the problems they're solving and the skill set you need to you need to have um changes through that evolution. And so we're also helping our engineers grow those skills and and helping give them feedback on how that growth is going.

36:06All right, thanks. Well, look, uh, that was super super nice to have you. Maybe maybe one last question. Do you have any any insight, any advice for aspiring engineer that's, you know, want to become a leader? I know you have you have been super successful like how how how can they become a polantist? That's tricky. I'd say, you know, career

36:28advice is hard. I think the first thing I'd say actually is um everything's easier when you're doing something you love. So um following what you're passionate at and what you're good at is a lot easier than like doubling down on your strengths and what you find over time that you're kind of uniquely good at in your environment can be a great way to progress in your career or

36:50progress in your own development um and have maximum impact in the environment you're in. So figuring out what that is and it's situational like you know you might be awesome at one thing. You know I'm very good at solving specific types of software optimization problems but somewhere along the way there were 10 other people who were good at that but none of them were good at let's say looking at how to build a system that

37:12crossed departmental boundaries but I was good at going and understanding the broader system and could solve problems that others weren't seeing. Um, so it's just an example where you need to be looking at what you're good at relative to the environment you're in and trying to maximize that. The second thing is don't optimize for the short term. Um, like I think it's really important to be looking at how you, you know, whether what you're doing is growing you. Um,

37:34you know, where do you want to be long term? Um, and often people make short-term choices. you know, they jump ship for a small increase in compensation and they do it three times in a row in three years and next thing you know you haven't built the knowledge and experience you need from having seen something all the way through its life cycle. So until you've seen a software system, you've designed it, you've seen

37:56it roll out, it's worked successfully, and then it's fallen over 3 years later due to the new requirements on the system and and thinking back how I would have changed or should have I changed anything or maybe it was fine. There's a there's a learning you get from sticking with a company, with a group, with a team, with a project for a particular period of time that is worth um that is worth something in your long-term trajectory that may not be apparent to

38:19you when you're when you're in on the journey. And so figuring out the right time when how long to stick with something and when to make a small change locally, within your team, across teams, across companies, or jumping continents. Um those are important choices to make. But don't be afraid to take a big crazy change too. I went and ran an office in Vietnam. I learned I did not want to be a general manager was

38:42one thing I learned from that experience. It was a wonderful experience. And then I also made the jump to come to Amsterdam. And it's been an amazing experience with Databricks. And so um I'm really glad for both of those opportunities. Even though one of them probably objectively on paper looks like it probably led to more than the other. Um both of them helped shape who I am.

39:05That was perfect. Thank you so much, Paul. Awesome. Well, thanks for having me on the show. Um, and uh, that was fun.