No-Code Data Pipelines with AI: Lakeflow Designer

0:00So, we've talked about why we're building like flow designer. Next, I want to show it to you. I'll walk you through building a simple example pipeline where I'm going to show you four things. First, how to import data from a spreadsheet and transform it using simple drag and drop. Second, I'll show you how to transform data by uh classifying it and translating it using AI with the Databricks assistant.

0:22Third, I'll show you how to transform data entirely by example before finally discussing how to deploy visual pipelines to prod. What we want to know is if support

0:44representatives with more experience have a higher rate of resolving customer calls. So that's the question. Does more experience mean more resolved calls? Now the employee tenure data I need is outside of Databricks in the spreadsheet. Now bringing spreadsheet data into designer super simple drag and drop immediately I see the data which I can work with like any other table. I've got the employee information here for

1:06each representative 0 to 3 months 3 to 6 months or more. Now let's combine it with the call transcripts that are already in UC. I'll add a data source here to the canvas. Now let me select the call transcripts table. Let me give this a name. We'll call this uh transcripts.

1:29Okay, let's see what we have here in the data preview. At the bottom of the screen, we see each row has a transcript, an employee ID, and a duration of the call. Now, this isn't exactly the data that I need. First, I need to know whether the support rep actually resolved the customer issue. And second, I see there's several different languages here. So, I'm going to ask the assistant here. translate transcripts label as resolved or not resolved

1:54and compute percentage by employee. All right, the assistant goes off. Thanks for a few seconds and we see two new operators appear here on the canvas. A transformation and an aggregation. Let's see what the assistant did here. The first step is translation and classification. I see the translated results over here. They look reasonable and we've got a classification of

2:16whether or not the support call was resolved. Now, what's happening here under the covers? The assistant uses our new built-in AI SQL functions to translate and classify these transcripts. But I didn't need to know any of the details about those functions. All right. Well, now let's look at the second operator counting the resolved. I've got the percentage of resolved cases down here, but the numbers look a little strange. I see 1%

2:383% 1%. The support reps all look terrible. I know it's not that bad. Now, what happened here? Well, as I'm sure everyone in the audience knows after after using Genai tools for a few years, they aren't perfect. Mistakes happen. And so, we're building LakeFlow Designer to make it easier to recognize and correct those mistakes. One example is what we see here, showing each step of a pipeline with its input and its output

3:01so that you can spot check suggestions. Another example is operator summaries. If I hover over this suggestion here, I can see a generated description of the operator. That summary here is percentage of all transcripts resolved by that employee. So, we can see what's wrong. This operator is calculating the percentage of all calls that a particular employee resolved. We can

3:23also see that in the SQL shown on the right. Now, language models are smart, but English is also very ambiguous. So, let's change this prompt. I'm going to add over their own calls. percent of by employee over their own calls. All right, again the assistant thinks and the percentage of resolved cases here looks a lot more reasonable. This data looks pretty good to me. I see 25% 45.

3:45This looks good. All right, let's accept this. Okay, so we've got our percentage resolved cases per employee. Now we're ready to join this with the tenure data that we brought in earlier from our spreadsheet. Remember that's the operator at the top left here. So, let's move this a bit closer to our percentage resolved operator and type join support rep data with percentage resolved.

4:09Okay, few more seconds of thinking. Now, we've got a join operator that combines these two. We see here the tenure information along with the employee ID and resolved cases. All right, this looks pretty good. I'm going to accept this. So, we've got our employees bucketed now by time and roll, but we need to tidy this up and look at the aggregated data. I've got a lot of extra columns and I need to aggregate by tenure. I know I'm going to need to present this data. Basically, I need to

4:32fill out a slide that looks like this. This is a situation where I've got an example of what I want the data to look like. In these cases, it can be easier to show the assistant what we want to do rather than to describe it. Let's see how that works here. Let's start with a simple screenshot of my slide. Okay, very good. Now I'll switch back to designer and I'm going to add a new

4:55operator transform by example. Again, this lets us show the assistant what we want with data or here even a screenshot. All right, let's attach our screenshot. All right, and here we go. The assistant goes off, thinks for a few seconds, and fills in the data that's shown in the image. And here we do indeed see longer tenur employees, thank you. Thank you.

5:17They have a higher support case resolution rate. Six months it's 80% new users 20%. Figured all this out with no code. But now what about productionization? I haven't told you anything about this. I don't need to show it to you because Bal already did under the covers. Pipelines in designer are completely standard lakeflow pipelines. All of the features that Bal

5:39showed you earlier apply here as well. If you need to collaborate with your data engineering team that prefers working in SQL, no problem at all. Edits you make to the SQL immediately apply to the canvas and vice versa. Let's select a few operators here to see what this looks like. As I select these, the corresponding SQL highlights. I'm selecting a few different operators here. The preview updates. It's all just

6:02SQL under the covers. Okay, here we go. A few more. Okay, very cool. Now, I'm going to switch browser tabs to show you the pipeline history and status page for a visual pipeline. Exactly the same as for any other LakeFlow pipeline. Again, no separate tool. It's all LakeFlow. All right. Very good. Let's go back to the slides and recap what we've seen. All right. First, sharing and collaboration

6:25between analysts and engineers. Super simple. You can edit pipelines visually or as code, whichever you prefer. Second, there's a built-in path to production. Designer is fully integrated with LakeFlow. There is no separate tool. And finally, we saw how designer uses AI throughout the product to boost productivity with features like transform by example. That is LakeFlow

6:49Designer, a visual editor that delivers production quality ETL for both business analysts and data engineers. No code required. All right, Designer will be available later this year. We're really excited for you to try it out. And with that, I'll welcome Alie back to the stage. Thank you.