Laktory: Terraform + dbt + Asset Bundles in One Framework

0:05hey today I have the pleasure to host Olivier Susi who gonna show us um a platform or a tool he developed on the top of data break but first let's let's get to know more Olivier hi hi youf uh thank you so

0:27much for having me um it's it's a it's a pleasure to be here to you know have that discussion with you um so a bit of introduction about myself um I I don't come from a data world I come from mechanical engineering I've been spending a good 10 to 12 years building uh flight simulation helicopter simulation to be to be exact but uh

0:50gradually I've uh you know moved towards more data Centric and and data Focus role and it kind of became my um my my passion and you know what what I've been driving to uh to build uh so I've worked with a few different companies and helping them either building a data platform or helping them with their data

1:12engineering work and as I was doing it I I found that there was a a need of you know some kind of framework that would help these companies to build their a data platform or a Lakehouse more efficiently so that's um essentially why I decided to start building the framework elory that I'm going to be

1:34presenting today um and you know it's it's it's an open source project that I'll be super happy to discuss and essentially I want to you know bring your attention to the existence of that framework so if anybody is interested in in using it uh then I'll be super happy to uh you know take them to a guided tour of some sort yeah and I know you

1:57explained this but what was the reason being behind developing this open source framework like leory if I'm not wrong yeah Lory that that's correct uh so essentially I've worked with a a crypto exchange company where uh you know I had to um put data pipeline into production and I wanted to make sure that these data pipeline were kind of uh

2:21tested uh they were governed and that I had the ability of deploying the same pipeline in a test environment run some test and then deploy them into um you know some kind of production environment so while Databricks offer plenty of feature to run data Pipeline and to have very efficient Computing uh it didn't

2:43have anything for kind of managing uh that process of developing deploying and maintaining data pipeline so that that's essentially where the idea come came from so I started building uh some kind of a framewor work internally and then I started to realize that a lot of companies are essentially doing the same thing uh in their uh own

3:08businesses uh so I thought it might be a good idea to actually uh you know extend the framework uh that I was developing make sure it's generic enough to suit everybody's needs and to essentially deploy it to the open source community so you know more businesses could could use it and my intention is really to build a community around it so so uh

3:30business can uh use it but mostly can contribute to it and you know grow something that's going to be very useful for many companies and you know we can essentially uh walk that journey together so that was the main motivation behind uh the development of this uh this framework okay that's that that's cool

3:54and I think if I'm not wrong like some parts may be overlapping with asset bundles I don't know if you're familiar with asset bundles but they can help like just deploy like of course the workflows and pipelines like DLT or jobs I think yours covers much more things than than acid bundles yeah that that's

4:17entirely correct and it's kind of a interesting story because I I started building this framework before asset bundles was actually released uh so for me it was kind of uh very interesting to see that dat rck was working on something similar but different at the same time and my offering like you said has some overlap with asset bundles uh

4:41but is also different in some ways so uh the main difference is that lacory support any uh resources that is available in Databricks so if you're talking about uh cluster warehouses Secrets catalog met store external location so any kind of component that you can deploy uh typically through

5:03terraform you can deploy it through lactor while uh asset bundles is more focused on on workflows uh I know that it has evolved in the last few weeks and few months so more component have been supported uh but the approach is very similar in in both cases it's a it's a yamal base uh configuration uh that you

5:27use to um you know deploy to Databricks um also in the case of lacory I I essentially support two different backend for the deployment so while asset bundles only support terraform lacter support both terraform and pulumi so if your if your business is using pumi then it's it's it's kind of interesting because you can integrate it

5:49in the same ecosystem you have uh you know the the same platform for for the deployment so you you you have the options uh but I would say that the key differentiator between lacory and asset bundles is lacory as an internal uh ETL pipeline model so not only you can Define your resources but you can also Define your data transformation and your

6:12your data asset and that's really the the big differentiator between the two and that's what I was trying to accomplish with lactor is the ability of not not only defining your resources but also uh defining your data transformation in in a single place so essentially what lacory is about is it's

6:34a proposition for replacing potentially asset bundles um terraform and also DBT because you can actually Define your data transformation so instead of having to uh deal with three different Technologies and three different framework uh you can package everything into uh you know a single solution so it

6:57makes it a lot simpler to you know operate your your data platform and something that I've seen in in many businesses is that typically you're going to have two group you're going to have a group that is more specialized in uh devops that will manage the infrastructure but also uh the deployment of unity catalog the uh

7:19configuration of the workspaces through uh terraform and you have a second group more data analyst data engineering or data uh data analytics uh Focus that will manage the workflows through asset bundles and to some extent it kind of creates some friction because you have to you know make sure that these two teams communicate very well and

7:43sometimes you have dependencies between the two and it's not as efficient as it as it could be uh so with Lory you have the ability of deploying everything from the same framework and because you don't need to understand terraform it makes it very accessible for uh data engineer and data analyst to actually manage not only their workflows but also to manage Unity

8:07Cog to manage the grant uh the secret and anything to to run your your louse and can you show us like just an overview of on how Lory looks like like is it yaml based and how do you feel the component to like an example how you deploy um maybe a workflow yeah so let

8:30me uh share something with you um that will kind of illustrate a little bit how lacory works and how it can be used um so I was saying uh you know lacory is really uh data ups and also a data frame Centric ETL framework so uh as opposed to DBT

8:52which is very SQL base uh lact is mainly built around uh data frame API so um as I was mentioning like why did I develop it is really about you know scaling up and deploying uh Databricks uh govern pipeline to production is typically really hard uh also working as a multi-user team uh that is also data Ops

9:16friendly is non-trivial and you know managing uh a lot of third party tools can be overwhelming for for a team so um in a nutshell lact is really about simplifying data um simplifying your ETL and as I was mentioning before um it's really about you know uh potentially

9:43sorry potentially um you know replacing asset bundles terraform and and DBT in a in a in a single solution um and you can see over here and I I'll go into more details about the you know the the Declaration into into yaml files um this is an example of a

10:05pipeline but uh in in terms of of architecture uh this is really how lacory works you have uh you know yo file that you input into the system it's going to do some uh validation and then um it will build your your your data Pipeline and then it will get deployed

10:27onto uh your your your data or space and the basic concept behind uh the data pipeline is you define a collection of node uh with a source a Transformer and a sync uh so your Source can be anything from you know a a volume on on Databricks it could be a data table it could be some some some files on on a pth and then your Transformer is

10:51really where we Define your data transformation so it could be um you know SQL transformation it could be spark uh uh function that you're calling and your sync is essentially where you want to write this uh this output so again to a volume to some kind of unity catalog table you have many options that are are supported so how does it

11:13actually work so let's look at uh an example um so as I was mentioning you can declare your entire lake house with Factory uh so your main entry point is uh a stack yaml file in which you're going to be defining um you know your collection of resources uh so it can range from catalog to secret warehouses

11:35notebooks um and the um you know the the flagship feature of Lory your actual pipelines and you know you have couple of options around this like I was mentioning you can Define your back end that that you want to use for your deployment so in this case we're using terraform but pumi is also available uh you have the option of defining multiple environments so if you want to deploy

11:58the same stack to multiple workspaces and have uh you know very specific configuration so very typical example is when you deploy to a a test environment and you know you don't want to run your workflow with the same schedule as you would do in your production environment then you have have the ability of reconfiguring your schedule um for each

12:22of the environment then you have the ability of finding variables um so like I was saying it's it's a very similar concept to bundles um so when you when you define the resource um it's essentially using terraform as a back end which also rely on you know datab bricks uh rest API so the name of the properties will be the same that you're

12:44already familiar with there's really nothing new to learn um the only difference is that you also have the ability of defining your Access Control within the resource itself so we have an example here where we Define warehouse and we can Define the access control directly in the definition of the resource which is really convenient because if you were to do that with

13:07terraform uh you know it's it's a lot of resource to Define you have to create links uh between the different IDs and it's much more involving so this is one example of a resource you also have the ability of defining a pipeline so uh this is really the key feature of lacory uh so you define your pipeline and you can Define and orchestrator so

13:30the orchestrator will be essentially what will be running the pipeline So currently you can deploy it as a data Bri job but you can also deploy it as a DLT table which is super interesting because um some user or or some businesses are kind of reluctant of using DT even if in my opinion it's a

13:52very good uh you know ETL framework uh so with this approach you have the ability of you know using the LT or not and all you have to do is change your orchestrator so it's a good way of trying it and and see if it uh if it fit you need without actually locking yourself to using that that technology um so once you've decided on

14:14your orchestrator all you have to do is Define a list of nodes which are essentially your your data tables so here we have an example where we're dealing with stock prices so your bronze table if we're following The Medallion architecture um we're going to be uh reading the data from a a volume um and we're going to be reading it as a string

14:36so it means that you can actually uh use uh database autol loader in the back end just by specifying this this option and then you define where you want to write it so in this case we're going to be writing to Unity catalog as as a table name and you have multiple you know write mode that are supported um in the case of the silver table it's kind of

14:58the same story story you define a source but in this case it's the previous node that you've defined H you also Define your your your sync you also have the ability of defining multiple sync um and I'll explain why in a minute but uh here the main difference is that you have a Transformer um and I was as I was explaining before the Transformer is

15:20essentially a a daisy chain of transformation and you can mix and match SQL expression and Spar function so this is really really powerful uh because it really helps you use the Best of Both Worlds whenever it it better fits your you need so for example here uh the First Transformation is just a select on

15:42the bronze table where you're going to be selecting your columns and then uh you're calling the drop duplicates method from spark um and then you essentially provide the uh the argument to that function and you can daisy chain as many operation as you want and you can also provide your own custom spark transformation so really really powerful in terms of what you can do you also

16:06have the ability of defining expectation so if you have um you know constraint that you want to impose on on your data set in terms of values for a given column uh you can also uh Define expectation in terms of aggregation so if you want the average value of a column to be above a certain value or if

16:30you want to count the number of column it's also possible and another great feature is if you define these expectation and Define an action called quarantine you can Al actually store these results into a quarantine table so you don't have anything else to do you just uh create a sync where you specify that you want to use a quarantine

16:53results so it really helps um you know putting together some some some quality gug and having the ability of exploring the data afterwards and like I was mentioning earlier all of that is kind of compatible with both data brick job and DLT so you can deploy it into uh you know both uh orchestrator depending on

17:16which one you you want to use so once you have your um data pipelines and all your resource um created uh you can use lacory CLI to do your deployment so you're just going to call Lory deoy and the name of the environment and it's going to use her form the backend so the the feedback

17:38that you're going to receive from the command prompt is exactly the same as you would from terraform um and you can you can use any kind of terraform options that that you want to use and once it's deploy um you have a few options for actually running your workflows and and your data Pipeline and one of the most most interesting one in

18:00my opinion is the ability of developing Lo locally using spark connect so um you can create uh a script a local script uh that's going to essentially run your pipeline so what it means is that you don't actually need to deploy anything to Databricks you can configure your yo file you can execute your pipeline and you're going to be able to explore the r

18:23resulting data frame in your favorite ID whether it's uh vs code or or py Chon or whatever you're using and you're going to see exactly what's happening with your data transformation and you can go back and forth and update your yo file and so it's very very efficient to uh you know develop your transformation to apply them see the result and make sure that they work before actually deploying

18:46it and once they're deployed um so this is an example where it has been deployed as a DLT uh table so very typical you have you know a few tables that views are also supported some of them are streaming some of them are not um and you like I was saying if you change the

19:08orchestrator you're going to have the ability of deploying exactly the same pipeline as a as a Databricks job and it's going to look very similar because each task is going to be uh about the creation of a of a single table so you're going to have a very similar field to the Delta life table and you also have the ability um from Lory to run these uh workflows so what it means

19:32it has very good support for cicd so um you know in a in a typical cicd framework you can Define your deployment and then run some of your workflow to make sure that they work before uh they're actually deployed to your production environment so you have this this ability of you know running your workflows with with

19:55Lory um so in a nutshell that that's really what lact is about so it's really about simplifying uh data Ops and and ETL practice um it really helps to accelerate and you know ship value much quicker and more reliably in into production environment and it's reducing the cost because uh developing those

20:17pipelines is much more efficient especially with the uh ability of do the uh the local development and and the local testing it also embrace you know the open source uh that is so close to Databricks with the you know Delta spark and all these other great components that are open source and it's

20:39also um a data braks technology partner so the company that I uh started o which is producing the software is a data R partner so it's a solution that is officially uh recognized by by Databricks um and yeah so in a nutshell that that's what uh Lector is about so uh um

21:02I I really believe it's really helpful for for for many businesses out there and I would like the community to grow and more user to use it and to contribute to it yeah and I think it's looks looks amazing and as you mentioned to me last time like everyone can contribute to this project it's how it's howed done GitHub so they can add some

21:25modifications suggestions maybe open issues in case but I see like it's super easy to get started with the with with leory and to be honest it's super great I will add of course the uh link to Au Cube and also the link to the to the project and if I'm not wrong I think you also

21:47developed a python Library if I'm not wrong yeah well laary is a python Library uh so it's a it's you can install it with a with a pip install which is also going to install the uh the CLI and you know what I've shown today is kind of the uh Databricks Focus aspect of lacory uh but eventually

22:11lacory is going to go far beyond this this ecosystem uh because what i't mentioned is lact is supporting multiple um data frame engine So currently it supports spark but it also support uh supports poers so if someone wants to build something entirely locally and run

22:34it locally they can with lacory because they can run it with uh with polers uh they can Define essentially the same data transformation but instead of using spark it's going to be using polers and this is all done through through python obviously um if they want to switch to spark then they all all they have to do essentially is to uh change the uh the

22:56back end for the data frame and then it's going to be using spark and using spark doesn't necessarily mean using uh Databricks it means that if you have like a local installation of spark or a spark installation on a cluster you can still use lacory obviously typically I would recommend to use Databricks because it offers so many more feature

23:19than you would have on your own uh but the idea is to be able to scale to the size of your problem so let's say that you start with a very small data set you have some exploration uh you want to keep the cost to minimum then you can essentially build some kind of proof of concept locally using poers or using a local spark installation and once you

23:41have something that works and you know that it's going to be uh kind of almost production ready then you can deploy it to datab bricks and use the full scalability of the platform and use uh Unity catalog to distribute your data so that's really the idea behind it is to be able to really scale your operation based on the size of your data set uh

24:03because sometimes you know we use Spark by default because it's kind of the engine that is uh you know available in dat bricks but it's not always the optimal choice so now you have that skill video option and you know you don't have to do any kind of extra work to move from one to the others because what we've seen is you know people working for example with pandas

24:26to do some some kind of analysis and it works great but when it comes time to deploy the same data transformation to data brakes it doesn't work because obviously uh pendas is not uh able to work on on multiple node and it's not really uh you know spark compatible so with lacory you can Define the same transformation locally and the one that

24:48are going to be using remotely on on spark so you have the ability of switching between the two very uh very easily so it's really up to the user to decide how they want to use it um they can use uh yam Mo configuration because it kind of standardize the process but they also have the uh the ability of building everything directly into python without having to write H

25:11file they can you know instantiate exactly the same classes do exactly the same kind of operation they can launch uh job remotely on on Databricks so um really up to the user preference that was crystal clear um great do you have anything else to add before we close the session um well I want to mention that

25:36uh you know working with data bres has been an amazing journey for for me uh you know when I I started building data platform I've kind of looked at the different options and uh for me data braas was kind of the most uh featured um platform so it was uh it was a Natural Choice to go with so if

25:58anyone in the community is using Databricks I encourage you to to look up for for lacory um and I I I think where it's really interesting is I I haven't seen many option out there that are data frame Centric most data transformation or SQL based and they don't really support spark transformation uh so if you're looking

26:20for a solution that will really be more compatible with spark or with data frame API um you should definitely uh you know have a look um I'm always available for for question if anyone is interested in a in a more extensive demo or if they have question I also have a slack channel available that they can they can reach out to I've been working with a

26:42few companies already uh you know the solution has been deployed into production and I've received really good feedback so it's really about getting the word out there I think the solution work it's just to get more adoption and then then and hopefully uh you know we'll be able to grow that solution together that's

27:04perfect thank you thank you so much for for this uh wonderful presentation and for those who are watching this recording all the resources will be added in the description of the video thank you Olivier thank you so much have a nice day and see you soon