Enterprise Data Ingestion & Orchestration with Lakeflow

0:00All right, it's time to show you a demo. I'm going to show you three things. I'm going to show you how to orchestrate production workflows, easily ingest enterprise data, and finally efficiently and reshape and clean it using LakeFlow. All right, let's get to the demo. All right, to show you what LakeFlow can do for you in the real world, I built an application for a major bank, which you can see here. It uses synthetic data.

0:23This is a PC, but it's very powerful. Uh with this data we're creating hyperpersonalized per customer recommendations and this unlocks millions of dollars worth of value. But of course uh let's see the real magic here is in the data. So I want to take a look at that. So this is LakeFlow connect and it comes with a suite of connectors that make it easy to ingest data from Salesforce workday service now

0:47and I'm going to run out of breath here. Many many other systems and I'm going to show you how easy this is to use. Okay. So let's quickly click on that and this is Salesforce and as you can see it's just four steps. You can bring fresh clean data from multiple places including CRM into Databricks but I'm not going to actually ingest this because I've taken care of this using lakeflow connect. Uh let me go to the lakeflow job and let's see it's this tab

1:12I think. Great. So now remember Layflow jobs is what runs your production workload and this is the job and as you can see it's a DAG it's a it's a graph of work to be done and this is the Salesforce ingestion. I'm pulling in campaign data then I'm pulling in SQL server from SQL server I'm pulling in customer profiles and since this is a large uh bank we have a legacy ZOS mainframe and I'm ingesting data using

1:34the secure FTP connector. After that, once the data is in the lakehouse, what I want to do is I want to build a hyperpersonalization pipeline using LakeFlow declarative pipelines. And this just means I'm going to build per customer recommendations over a lot of data. And this is normally very very difficult and takes, you know, months and months. This is a branching operator in the job. I use this to check if the pipeline succeeds. If it fails, I send

1:57an alert and if it succeeds, then I update an AIBI dashboard. I write recommendation into Unity catalog. And then finally, I use a looping operator to loop over a set of virtual machines to update them as well. So my entire logic here is fully encapsulated inside a job and I can take care of everything in one place. Now I want to show you one really cool thing and let's take a look at this. Let's take a look at this tab. So lakeflow jobs offers deep monitoring

2:20and insights into your workloads. So I can see and control where both time and money are being spent here and I can optimize and I can sleep well uh without uh losing control over my work. So let's zoom in on this this hyperpersonalization pipeline which I built using LakeFlow declarative pipelines. So welcome to the new IDE for data engineering. So we built this so

2:42you can focus on the most important parts of data engineering. It has AI assistance, data exploration and debugging built right in and also production readiness. And this is the data flow graph. And what's happening here is data arrives on the left and it ends up in these tables, these gold tables which are materialized views. All right. And I can use this this graph to navigate and monitor the pipeline. All

3:04right. So what I'm going to show you here is that this final gold table for example contains the table that power my application. We have our per customer recommendations. So our customer service reps can give a great experience. So that's just the final output. And I can take a look at the other final gold table as well which has further recommendations and rebalancing opportunities. What I'm going to do now is I'm going to expand some of the panes of the IDE to show you what's going on

3:26over here. So I'm going to minimize the DAG here. I'm going to drag some things around a little bit to create some space. So this is the file tree and I use this to organize my code into bronze, silver, and gold layers. And I can organize the way I like. It's up to me. So you can see I did the medallion architecture here. And I added some tests as well. So the IDE includes testing a native testing framework. I write some code here. And this is the

3:48familiar DAG that I just showed you, the pipeline graph. And okay, let's take a look. So I can I can explore the through the DAG, but I can also have this purpose-built pane at the bottom. And in the PES at the bottom, we have the ability to view tables, performance, and so forth. All right, so let's take a look at the code behind one of these tables. And you're going to be a little surprised to see that there's actually

4:10very little code. And that is the power of declarative ETL. You're freed up to write your business logic without worrying about infrastructure. And of course, you can mix in SQL or you can also mix in Python in LakeFlow declarative pipelines. So I'm going to run this pipeline in a moment and I'm going to show you something really cool. This is powered by serverless compute for the best possible experience. And while the pipeline runs, I'm going to

4:33show you a quick demo of the of the uh database assistance. And I'm going to add some data quality checks. And I don't know how to write those. I'm just going to ask the assistant. And as you can see, the assistant added a valuable data quality check. And this is how I have high val high quality data. And finally, just to put it into production, it's as simple as hitting the schedule button. And with that, my pipeline is

4:55ready for CI/CD and DevOps. All right, so that's the demo. And let's switch back to slides. And so you see in the demo, I showed you how to orchestrate a production workflow, easily ingest enterprise data, and efficiently reshape and clean it. The IDE for data engineering is available for use today and LakeFlow is officially generally available and ready for the world's

5:17toughest workloads. Thank you.