Laktory
Laktory DataOps Framework
hey today I have the pleasure to host Olivier Susi who gonna show us um a platform or a tool he developed on the top of data break but first let's let's get to know more Olivier hi hi youf uh thank you so
much for having me um it's it's a it's a pleasure to be here to you know have that discussion with you um so a bit of introduction about myself um I I don't come from a data world I come from
mechanical engineering I've been spending a good 10 to 12 years building uh flight simulation helicopter simulation to be to be exact but uh gradually I've uh you know moved towards
more data Centric and and data Focus role and it kind of became my um my my passion and you know what what I've been driving to uh to build uh so I've worked with a few different companies and
helping them either building a data platform or helping them with their data engineering work and as I was doing it I I found that there was a a need of you know some kind of framework that would
help these companies to build their a data platform or a Lakehouse more efficiently so that's um essentially why I decided to start building the framework elory that I'm going to be
presenting today um and you know it's it's it's an open source project that I'll be super happy to discuss and essentially I want to you know bring your attention to the existence of that
framework so if anybody is interested in in using it uh then I'll be super happy to uh you know take them to a guided tour of of some sort yeah and I know you explained this but what was the reason
being behind developing this open source framework like leory if I'm not wrong yeah Lory that that's correct uh so essentially I've worked with a a crypto exchange company where uh you
know I had to um put data pipeline into production and I wanted to make sure that these data pipeline were kind of uh tested uh they were governed and that I had the ability of deploying the same
pipeline in a test environment run some test and then deploy them into um you know some kind of production environment so while data bricks offer plenty of feature to run data Pipeline and to have
very efficient Computing uh it didn't have anything for kind of managing uh that process of developing deploying and maintaining data pipeline so that that's essentially where the idea come came from so I started
building uh some kind of a framewor work internally and then I started to realize that a lot of companies are essentially doing the same thing uh in their uh own businesses uh so I thought it might be a
good idea to actually uh you know extend the framework uh that I was developing make sure it's generic enough to suit everybody's needs and to essentially deploy it to the open source community
so you know more businesses could could use it and my intention is really to build a community around it so so uh business can uh use it but mostly can contribute to it and you know grow
something that's going to be very useful for many companies and you know we can essentially uh walk that journey together so that was the main motivation behind uh the development of this uh
this framework okay that's that that's cool and I think if I'm not wrong like some parts may be overlapping with asset bundles I don't know if you're familiar with asset bundles but they can help
like just deploy like of course the workflows and pipelines like DLT or jobs I think yours covers much more things than than acid bundles yeah that that's entirely correct and it's kind of a
interesting story because I I started building this framework before asset bundles was actually released uh so for me it was kind of uh very interesting to see that dat rck was working on
something similar but different at the same time and my offering like you said has some overlap with asset bundles uh but is also different in some ways so uh the main difference is that lacory
support any uh resources that is available in data bricks so if you're talking about uh cluster warehouses Secrets catalog met store external location so any kind of component that
you can deploy uh typically through terraform you can deploy it through lactor while uh asset bundles is more focused on on workflows uh I know that it has evolved in the last few weeks and
few months so more component have been supported uh but the approach is very similar in in both cases it's a it's a yamal base uh configuration uh that you use to um you know deploy to to data
bricks um also in the case of of lacory I I essentially support two different backend for the deployment so while asset bundles only support terraform lacter support both terraform and pulumi
so if your if your business is using pumi then it's it's it's kind of interesting because you can integrate it in the same ecosystem you have uh you know the the the same platform for for
the deployment so you you you have the options uh but I would say that the key differentiator between lacory and asset bundles is lacory as an internal uh ETL pipeline model so not only you can
Define your resources but you can also Define your data transformation and your your data asset and that's really the the the big differentiator between the two and that's what I was trying to
accomplish with lactor is the ability of not not only defining your resources but also uh defining your data transformation in in a single place so essentially what lacory is about is it's
a proposition for replacing potentially asset bundles um terraform and also DBT because you can actually Define your data transformation so instead of having to uh deal with three different
Technologies and three different framework uh you can package everything into uh you know a single solution so it makes it a lot simpler to you know operate your your data platform and
something that I've seen in in many businesses is that typically you're going to have two group you're going to have a group that is more specialized in uh devops that will manage the
infrastructure but also uh the deployment of unity catalog the uh configuration of the workspaces through uh terraform and you have a second group more data analyst data engineering or
data uh data analytics uh Focus that will manage the workflows through asset bundles and to some extent it kind of creates some friction because you have to you know make sure that these two
teams communicate very well and sometimes you have dependencies between the two and it's not as efficient as it as it could be uh so with Lory you have the ability of deploying everything from
the same framework and because you don't need to understand terraform it makes it very accessible for uh data engineer and data analyst to actually manage not only their workflows but also to manage Unity
Cog to manage the grant uh the secret and anything to to to run your your louse and can you show us like just an overview of on how Lory looks like like is it yaml based and how do you feel the
component to like an example how you deploy um maybe a workflow yeah so let me uh share something with you um that will kind of illustrate a little bit how lacory works and how it can be used um so I was
saying uh you know lacory is really uh data ups and also a data frame Centric ETL framework so uh as opposed to DBT which is very SQL base uh lact is mainly built around uh data frame API so um as
I was mentioning like why did I develop it is really about you know scaling up and deploying uh data bricks uh govern pipeline to production is is typically really hard uh also working as a
multi-user team uh that is also data Ops friendly is non-trivial and you know managing uh a lot of third party tools can be overwhelming for for a team so um in a nutshell lact is really about
simplifying data um simplifying your ETL and as I was mentioning before um it's really about you know uh potentially sorry potentially um you know replacing asset bundles terraform and
and DBT in a in a in a single solution um and you can see over here and I I'll go into more details about the you know the the the Declaration into into yaml files um this is an example of a
pipeline but uh in in terms of of of architecture uh this is really how lacory works you have uh you know yo file that you input into the system it's going to do some uh validation and then
um it will build your your your data Pipeline and then it will get deployed onto uh your your your data or space and the basic concept behind uh the data pipeline is you define a collection of node uh with a source a
Transformer and a sync uh so your Source can be anything from you know a a volume on on data bricks it could be a data table it could be some some some files on on a pth and then your Transformer is
really where we Define your data transformation so it could be um you know SQL transformation it could be spark uh uh function that you're calling and your sync is essentially where you
want to write this uh this output so again to a volume to some kind of unity catalog table you have many options that are are supported so how does it actually work so let's look at uh an
example um so as I was mentioning you can declare your entire lake house with Factory uh so your main entry point is uh a stack yaml file in which you're going to be defining um you know your
collection of resources uh so it can range from catalog to secret warehouses notebooks um and the um you know the the the flagship feature of Lory your actual pipelines and you know you have couple
of options around this like I was mentioning you can Define your back end that that you want to use for your deployment so in this case we're using terraform but pumi is also available uh
you have the option of defining multiple environments so if you want to deploy the same stack to to multiple workspaces and have uh you know very specific configuration so very typical example is
when you deploy to a a test environment and you know you don't want to run your workflow with the same schedule as you would do in your production environment then you have have the ability of
reconfiguring your schedule um for each of the environment then you have the ability of finding variables um so like I was saying it's it's a very similar concept to to bundles um so when you
when you define the resource um it's essentially using terraform as a back end which also rely on you know datab bricks uh rest API so the name of the properties will be the same that you're
already familiar with there's really nothing new to to learn um the only difference is that you also have the ability of defining your Access Control within the resource itself so we have an
example here where we Define warehouse and we can Define the access control directly in the definition of the resource which is really convenient because if you were to do that with
terraform uh you know it's it's a lot of resource to Define you have to create links uh between the different IDs and it's much more involving so this is one example of a resource you also have the
ability of defining a pipeline so uh this is really the the key feature of of lacory uh so you define your pipeline and you can Define and orchestrator so the orchestrator will be essentially
what will be running the pipeline So currently you can deploy it as a data Bri job but you can also deploy it as a DLT table which is super interesting because um some user or or some
businesses are kind of reluctant of of using DT even if in my opinion it's a very good uh you know ETL framework uh so with this approach you have the ability of you know using the LT or not and all you have to
do is change your orchestrator so it's a good way of trying it and and see if it uh if it fit you need without actually locking yourself to using that that technology um so once you've decided on
your orchestrator all you have to do is Define a list of nodes which are essentially your your data tables so here we have an example where we're dealing with stock prices so your bronze
table if we're following The Medallion architecture um we're going to be uh reading the data from a a volume um and we're going to be reading it as a string so it means that you can actually uh use
uh database autol loader in the back end just by specifying this this option and then you define where you want to write it so in this case we're going to be writing to Unity catalog as as a table
name and you have multiple you know write mode that are supported um in the case of the silver table it's kind of the same story story you define a source but in this case it's the previous node
that you've defined H you also Define your your your sync you also have the ability of defining multiple sync um and I'll explain why in a minute but uh here the main difference is that you have a
Transformer um and I was as I was explaining before the Transformer is essentially a a daisy chain of transformation and you can mix and match SQL expression and Spar function so this
is really really powerful uh because it really helps you use the Best of Both Worlds whenever it it better fits your you need so for example here uh the First Transformation is just a select on
the bronze table where you're going to be selecting your columns and then uh you're calling the drop duplicates method from spark um and then you essentially provide the uh the argument
to that function and you can daisy chain as many operation as you want and you can also provide your own custom spark transformation so really really powerful in terms of of what you can do you also
have the ability of defining expectation so if you have um you know constraint that you want to impose on on your data set in terms of values for a given column uh you can also uh Define
expectation in terms of aggregation so if you want the average value of a column to be above a certain value or if you want to count the number of column it's also possible and another great
feature is if you define these expectation and Define an action called quarantine you can Al actually store these results into a quarantine table so you don't have anything else to do you
just uh create a sync where you specify that you want to use a quarantine results so it really helps um you know putting together some some some quality gug and having the ability of exploring
the data afterwards and like I was mentioning earlier all of that is kind of compatible with both data brick job and DLT so you can deploy it into uh you know both uh orchestrator depending on
which one you you want to use so once you have your um data pipelines and all your resource um created uh you can use lacory CLI to do your deployment so you're just going to call Lory deoy and the name of the
environment and it's going to use her form the backend so the the the feedback that you're going to receive from the command prompt is exactly the same as you would from terraform um and you can
you can use any kind of terraform options that that you want to use and once it's deploy um you have a few options for actually running your workflows and and your data Pipeline and
one of the most most interesting one in my opinion is the ability of developing Lo locally using spark connect so um you can create uh a script a local script uh that's going to essentially run your
pipeline so what it means is that you don't actually need to deploy anything to data bricks you can configure your yo file you can execute your pipeline and you're going to be able to explore the r
resulting data frame in your favorite ID whether it's uh vs code or or py Chon or whatever you're using and you're going to see exactly what's happening with your data transformation and you can go
back and forth and update your yo file and so it's very very efficient to uh you know develop your transformation to apply them see the result and make sure that they work before actually deploying
it and once they're deployed um so this is an example where it has been deployed as a DLT uh table so very typical you have you know a few tables that views are also supported some of them are streaming
some of them are not um and you like I was saying if you change the orchestrator you're going to have the ability of deploying exactly the same pipeline as a as a data bricks job and
it's going to look very similar because each task is going to be uh about the creation of a of a single table so you're going to have a very similar field to the Delta life table and you
also have the ability um from Lory to run these uh workflows so what it means it has very good support for cicd so um you know in a in a typical cicd framework you can Define your deployment
and then run some of your workflow to make sure that they work before uh they're actually deployed to your production environment so you have this this ability of you know running your
workflows with with Lory um so in a nutshell that that's really what lact is about so it's really about simplifying uh data Ops and and ETL practice um it really helps to accelerate and you know ship value much
quicker and more reliably in into production environment and it's reducing the cost because uh developing those pipelines is much more efficient especially with the uh ability of do the
uh the local development and and the local testing it also embrace you know the open source uh that is so close to data bricks with the you know Delta spark and all these other great
components that are open source and it's also um a data braks technology partner so the company that I uh started o which is producing the software is a data R partner so it's a solution that is
officially uh recognized by by data bricks um and yeah so in a nutshell that that's what uh Lector is about so uh um I I really believe it's really helpful for for for many businesses out there
and I would like the community to grow and more user to use it and to contribute to to it yeah and I think it's looks looks amazing and as you mentioned to me last time like everyone can contribute to
this project it's how it's howed done GitHub so they can add some modifications suggestions maybe open issues in case but I see like it's super easy to to get started with the with with leory
and to be honest it's super great I will add of course the uh link to Au Cube and also the link to the to the project and if I'm not wrong I think you also developed a python Library if I'm not
wrong yeah well laary is a python Library uh so it's a it's you can install it with a with a pip install which is also going to install the uh the the CLI and you know what I've shown
today is kind of the uh data bricks Focus aspect of lacory uh but eventually lacory is going to go far beyond this this ecosystem uh because what i't mentioned is lact is supporting multiple
um data frame engine So currently it supports spark but it also support uh supports poers so if someone wants to build something entirely locally and run it locally they can with lacory because
they can run it with uh with polers uh they can Define essentially the same data transformation but instead of using spark it's going to be using polers and this is all done through through python
obviously um if they want to switch to spark then they all all they have to do essentially is to uh change the uh the back end for the data frame and then it's going to be using spark
and using spark doesn't necessarily mean using uh data bricks it means that if you have like a local installation of of spark or a spark installation on a cluster you can still use lacory obviously typically
I would recommend to use data bricks because it offers so many more feature than you would have on your own uh but the idea is to be able to scale to the size of your problem so let's say that
you start with a very small data set you have some exploration uh you want to keep the cost to minimum then you can essentially build some kind of proof of concept locally using poers or using a
local spark installation and once you have something that works and you know that it's going to be uh kind of almost production ready then you can deploy it to datab bricks and use the full
scalability of the platform and use uh Unity catalog to distribute your data so that's really the idea behind it is to be able to really scale your operation based on the size of your data set uh
because sometimes you know we we use Spark by default because it's kind of the engine that is uh you know available in dat bricks but it's not always the optimal choice so now you have that
skill video option and you know you don't have to to do any kind of extra work to move from one to the others because what we've seen is you know people working for example with pandas
to do some some kind of analysis and it works great but when it comes time to deploy the same data transformation to data brakes it doesn't work because obviously uh pendas is not uh able to
work on on multiple node and it's not really uh you know spark compatible so with lacory you can Define the same transformation locally and the one that are going to be using remotely on on
spark so you have the ability of switching between the two very uh very easily so it's really up to the user to decide how they want to use it um they can use uh yam Mo configuration
because it kind of standardize the process but they also have the uh the ability of building everything directly into python without having to to write H file they can you know instantiate
exactly the same classes do exactly the same kind of operation they can launch uh job remotely on on data bricks so um really up to the user preference that was crystal clear um great do you have anything else to add
before we close the session um well I want to mention that uh you know working with data bres has been an amazing journey for for me uh you know when I I started building data platform I've kind of looked at the
different options and uh for me data braas was kind of the the most uh featured um platform so it was uh it was a Natural Choice to to go with so if anyone in the community is using data
bricks I encourage you to to to look up for for lacory um and I I I think where it's really interesting is I I haven't seen many option out there that are data frame Centric most data transformation
or SQL based and they don't really support spark transformation uh so if you're looking for a solution that will really be more compatible with spark or with data frame API um you should definitely uh you know
have a look um I'm always available for for question if anyone is interested in a in a more extensive demo or if they have question I also have a slack channel available that they can they can
reach out to I've been working with a few companies already uh you know the solution has been deployed into production and I've received really good feedback so it's really about getting
the the word out there I think the solution work it's just to to get more adoption and then then and hopefully uh you know we'll be able to to grow that solution together that's perfect thank
you thank you so much for for this uh wonderful presentation and for those who are watching this recording all the resources will be added in the description of the the video thank you
Olivier thank you so much have a nice day and see you soon