Data Warehouse Migration: 4-Step Databricks Strategy

0:04hi folks today we're going to discuss a topic different from what we used to discuss and we have a guest hi lauro Hi youself how are you doing good very good so tell me more about you and your role at datab braas so um my role at datab brakes is to be uh the migration

0:28specialist for uh em and especially for data warehouse Migra migrations previously to this world I have been a Solutions architect for more than two years and a half dealing with the biggest customers in EMA um and here I am a migration specialist for all this data warehouse stuff cool I'm I'm very curious I've

0:51seen a lot of customers having Legacy data warehouse uh on premise and I was curious to know like about more about the steps that need to be followed to have a successful migrations well first of all we have to understand why the data warehousing and the data warehousing is the data

1:14warehouse sorry is usually serving the business so we have to clearly understand the business of our customer and why the database the data warehouse serve the business so assessing all the business case is a key element of the process then we will um analyze assess the current platform and more on the

1:36technical uh side of the of the of the migration that means that we will look at the current environment the different the current uh architecture which is implemented uh the current data model because data modeling is key for data warehousing and of course in datab break we can Implement all the various data models

1:59that exist uh as of today so um knowing the the current platform is the first uh step of a migration process then we will analyze the target uh the target environment and so the target environment of course is data brakes that means that we will serve also

2:21certain types and certain number of business case into Data Brak and maybe probably we will had some new uh modern use case like real time streaming to help our customers to implement real time uh decision making uh we will probably implement some new use case

2:43built on top of machine learning and even gen and so once we will have defined what will run in the Target environment and knowing the the starting point of the migration we will be able to draw a way to transform the current environment to the

3:05Target environment that means that we will use some uh tools some um process to make the architecture evolve from a point to another another one uh so as I mentioned we will use some tools we will use some uh migration process that we will Define with our customer to transform their current environment

3:28smoothly to uh that brics interesting so just to summarize you need to understand the business context then understand the logic or the code to know how to refactor this code to be executed on the Databricks platform and according to you which part is mainly complex is it the

3:52understanding the business or the maybe the conversion of the code or the refactoring of the code well understanding the business is key as I mentioned but every one of us is able to discuss with the customer be curious of their environment and have a a very interesting discussion to understand why they are using such an environment

4:14to uh to implement their decision-based system um transforming is quite a bit complex because uh usually all these data wees are implemented in uh relational database uh system um and so we have to convert uh all the ETL

4:36process all the data the the table the model definition to data braks so that's not the most complex part of the of the transformation process but we also have to uh care about the different ETL process how they are implemented uh is the Cur pform using a third party tool for itl process uh how

4:58we can convert uh this from a stored procedure to data braks for example and so we have to think about this and build the the new way to implement the decision system into data braks and according to you uh I've been reading some medium past on some customers who did migr and often they

5:22try to replicate what they had in their legacy system in the new system like I can talk about like for example store procedures some of some of them for example in we don't have S procedures now it's on the way but it is always a good thing to replicate what we had or what the customer had or maybe think about new ways of uh making the platform

5:47more modern yes well speaking technically speaking you have to care about this transformation let's take an example imagine you have a star schema on uh um Legacy uh database Solutions uh and in this database you can Implement some decision based uh uh workflows but you can also

6:11Implement some oltp workloads that means that the solution the rdbms has a set of different tools or object that will be able to optimize the workload by putting some indexes some partitioning strategies and that kind of uh techniques to uh to to make the the workload efficient if you move

6:35to data braks we don't have indexes we have uh partitioning strategies and we have more we have clustering uh on on Delta for example and the way you will partition your data on the source platform and the way you will Implement a clustering uh strategy on data risk can be completely different and so for example you have to

6:58care about this uh um and uh um yes one more question for you we announced uh at the data ni Summit several uh new features for the data warehousing it includes uh Auto liquid Auto stats predict predictive IO

7:232.0 uh yeah so yeah which one is your favorite and which one you would you want to test first like according to you well um I like the the auto liquid clustering um I'm very uh uh impatient to to test it out because

7:46each time you will have for example a s schema it will be uh in very key to have a clean partitioning as I mentioned this before and so having the auto liquid clustering will help us to depending on the work and depending all the the analysis that we uh will execute on our uh on our side to determine what

8:09will be the best uh uh clustering strategy and make it happen automatically will definitely is the way to implement uh a data wareing strategy on data breaks yeah that that makes sense and just to give more context Databricks introduced first liquid clustering where we can add a clustering here as lauron

8:33have been mentioning then they did introduce predictive optimization that will go and run the optimize and vacuum which are some let's say overhead that was on the top of data engineers then they introduce Auto stats so as a data as a data engineer used to run analyze to capture statistics like mean Max for every column which can be

8:57helpful for the spark execution plan and now on the top of it besides capturing statistics we're going to understand the P the query pattern like how often you run I don't know in new works like uh select start from customers where countes equal to France or you query by by City And based on the number of queries like we will adapt the

9:19clustering key to be optimized for your queries and improve them which means if now you're quering by year by year and by country into three months you start uh filtering by year and City your layer your clustering key will change automatically without with with the massive impact which is the performance and I I'm pretty convinced

9:43that you know for all the existing platform the that makes years that they are implementing automatic strategies but now the future is on AI and I think that all the stuff around predictive iio and the way we will automatically change the the the clustering uh key the way we will uh automatically

10:05trigger a statistic uh collection and that kind of stuff is the future and the future and the future of datab breas is also part of it and it will drastically is the way we will use uh data brakes and a data warehousing uh strategy on top of data braks thank you Ron and stay tuned for

10:29part to very soon