Build Data Pipelines Without Code—Lakeflow Designer Demo

0:04Hey folks, today we have the pleasure to host Jason for the third time. We brought him I think 1 year ago maybe even more to talk about the notebook experience or the revamped notebook experience. Now he's back but with another topic but again it's still on the dev experience.

0:26Hi Jason. >> Hey Yosif, yeah thanks for having me. Great to be here. Yeah, my name is Jason for everybody. I'm a product manager here at Databricks and right now I'm focusing on Lake Flow Designer which is basically our low-code data prep solution directly on Databricks and uh it's hard to say this, I love notebooks a lot but I think Lake Flow Designer is the my favorite product I've ever worked on. So I'm excited to talk to y'all more

0:48about it today. So I've got some slides and a demo, should I just jump into that Yosif? So yeah, so I'm going to talk about Lake Flow Designer today which is a is a brand new product that we've released in Databricks recently. We just went to public preview and really the goal with Lake Flow Designer is helping your organization scale their ability to make data-driven decisions and what we mean by that is we want to allow everybody to be able to prepare

1:10data and then make decisions based on that data. So just to kind of start with some some background, right? Everything that you do on Databricks and with data depends on good data. Whether you're building AI models, whether you're building apps, whether you're building dashboards, you need to have good data that is a foundation for that. But the problem is that the people with the deepest understanding of the business are generally not the ones who

1:33can prepare good data. So in other words, the people who often need to make decisions based on the data are not the people who can prepare that and self-serve and create that themselves. So that creates the need for low-code self-service tools. And we're definitely not the first people to notice this need. There's a lot of other really really great tools in the market to help do this. For example, Alteryx

1:55Data IQ and there's also other like spreadsheet or other no-code tools Uh um like Excel and Google Sheets. But the challenge is that when you work with these existing low-code tools, the data work kind of happens in two separate worlds. So, on in one world, you have your Databricks environment or your central data platform, which is everything's governed, everything runs in production there, there's um yeah, just the thing things like this

2:18is really what's what's built for production, and you have kind of these low-code tools. They live outside of this governed environment. They're often on people's local machines um where people are doing things with spreadsheets. And so, this creates kind of a bit of conflict. The first is that you have tool sprawl. So, nobody likes to have a hundred different data tools um and so, it's not fun to have to have your central data platform and then go get different low-code tools. Um and

2:41when you do that, it creates some issues like with governance, for example, where you can't um govern the uh the tool use and the data that goes through these other low-code tools because they're not kind of in a central environment. It also creates duplication of your work. So, you might do one thing on a low-code tool that's on somebody's local machine. Um that gets done on multiple people's local machines, and it's maybe also done on the central data platform. Um and just generally, it can

3:04create inconsistency. The second big problem here is that you have trouble when you need to bring these things to production. So, you know, maybe I built out this workflow. I've got like 200 little boxes that I've dragged and connected together, and I do this awesome thing. Now, once I find this is useful, and I want to hand it to the central data engineering team to productionize it, the data engineering team gets it, they're like, "Whoa, there's like 200 boxes on the screen. How am I supposed

3:26to productionize this?" And that becomes um somewhat challenging to do. Um So, that's another challenge. And then the third is these tools, because they live outside of the central data platform, they live outside of Unity Catalog, and they don't have access to the rich metadata um that you could have if you built something directly in the central data platform. And then kind of undercutting all of this is uh we often see with these tools

3:49is there's expensive per user licensing. And so, before you even start, you have to pick which users you think will be worth it to give and pay this fee for, which also limits who can use these tools. So, our solution to this, our answer to this is Lakehouse Designer. It's no-code data preparation directly inside of Databricks, native to Databricks. And there's a couple things to just to

4:12call out here that are that are cool with it. One is it's drag-and-drop, so it's super easy to put together. It's AI-native, and so it uses Unity Catalog, uses all the meta data to be able to help you transform your data. And we built it from the ground up using AI. So, hopefully, as I'll show later, there's a lot of experiences that are very intuitive and very powerful because we leverage AI. And then everything is backed with production-ready code so that you can then go to production. So,

4:36ultimately, what this means is that you have a fully self-service, fully governed um capability directly in Databricks. There's no challenges with governance. Everything you do in it is is fully governed. You have a simple path to production as everything's backed in production-ready code, and it sits in your data platform already. If you want to take it and schedule it or put it as part of a a production job or a pipeline, you can do that really easily.

4:58And then, as I mentioned earlier, everything is AI-native and reviewable. And so, one of the challenges that we've been seeing is that LLMs are really great at generating responses and and answering questions, but often it's really hard to validate and trust and really know what was happening. Unless you're you're really good with code, you get a thousand lines of code, you're like, "Man, how do I understand what's going on?" The cool thing about Lakehouse Designer is it breaks things up into atomic operations,

5:21and you can see the flow of data, you can see step by step how it's changing to really understand what Genie code was building, for example. And then finally, this is all native to Databricks with consumption-based pricing. And so, again, you can enable all of your users to use this tool, and you don't have to pay a big fat fee up front to do that. You only pay for the data that you use, which is another great thing. And

5:43so, ultimately, we hope together this will help you scale your organization's ability to make data-driven decisions. Now, that's enough slides. Uh for those of you who know me, I usually go stick with short slides cuz I like to demo for the majority of the time. So, I'm going to jump over um into a demo right now. Um any comments or thoughts so far? You want to sit for or should I just jump into the demo? >> I think I think Databricks is really

6:08going towards this democratization of the data to business users. Um it all started with uh first with Genie. Uh AI BI Genie. Uh we added the dashboard. Now, recently Genie code. >> Right. >> The possibility to view or to consume

6:32all those assets using Databricks one. >> Right. >> And in addition, now we can also start everything from the ingestion. You can literally use um Databricks connect with the UI this connector to bring data to Unity catalog or govern through Unity catalog. And then you can take those tables and create your own orchestra

6:55create your own transformation or KPI uh using using of course Lake Flow Designer. And then out of those KPI, you can build your semantic layer using Genie code. And then you can bring this Genie code inside uh AI BI Genie. And you can also build own dashboard that you can consume through

7:17Databricks through Genie. >> Yeah, exactly. >> Databricks one. And of course, everything can be accessed in your mobile phone whether you have an iOS or Android and without writing a single line of code. And this is just amazing. >> Yes. Yeah. Yeah, we complete kind of the end-to-end journey with your data. And that's where I think Lake Flow Designer fits in here because previously, if you

7:40you to transform and prepare your data, right, you could do the dashboarding, you could do all that stuff, but if you wanted to prepare the data, that would require you could use Genie code, but it would create code that you couldn't really understand. So, we're kind of bridging that gap now and I'm giving you a low-code way to do that. So, I I totally agree, right? And and for those like, you know, 2 years ago Databricks might have been seen as a very technical platform that was only for data engineers. That is very different now.

8:02Um and I think as you'll see with Lake Flow Designer, this is another tool in your tool belt that will help anybody really. Anyone who can speak a language to be able to transform their data. Um So, let's uh let let's jump in. So, um to start, I guess let's just show how you get into Lake Flow Designer. So, the first thing you do is you go to the new menu and we have this new object here called visual data prep. And so, I can click to create a visual data prep.

8:25And then I get brought into the Lake Flow Designer um starting home screen. I actually have a a a file already created, which is in a good folder, so we'll we'll go ahead and use that, but I just want to show you how to get into that. And then from here, the first thing that we need to do is we need to add um data. We need to add add data that we can then transform, prepare, um and analyze. And so, there's a few different ways to do that and I do want to show you a few different ways cuz I think it will be

8:47helpful. The first is by creating a just a plain source operator. So, once I do that, I can say, "Hey, I can browse existing data that's in Unity Catalog. I can create a table from a file. I can drag and drop um from my local machine or I can even connect to external sources like Google Drive or SharePoint." For this, I'll just show you, you know, maybe I want to browse existing. I have some hotel properties data that I want to look at. And so, then I can click that and then in a

9:10matter of seconds, we should see a preview of the data. So, we can see all that really quickly um here. Now, we could just if I want to pull in some different tables, so if I wanted to, you know, I could just go over here and add another source operator and go through and do that. But actually, I think the way that is the fastest and maybe the easiest here is if I go over to the catalog explorer here on the left panel, I'll search for my name, Jason Messer,

9:33and then let's expand uh this list here. And then I can just drag file or drag tables directly in from that. So, I can say, "Okay, hotel bookings. I want to add this. I'll add cities." >> That's cool. I did I I wasn't aware of this one. >> Yeah, yeah. A lot of people aren't, which is why I wanted to show that. Um, so you can really easily, yeah, pull pull your tables in uh, directly from the side panel, which which helps a lot.

9:56Um, another way that you can you can pull data in is just from your local machine, you can just drag and drop. So, um, you all can't see I don't think you can see my desktop, but I have a file here in my finder and I'm dragging it onto the canvas. And then it just says, "Oh, hey, you want to upload this hotels reviews?" Um, and then I can just say where I want to put it. I just hit overwrite because I've already I've done this demo before and so have this file exist uh, that already exists. And then

10:18I hit upload here and it just pops in. And so, voila. I was able to basically bring in a CSV just like that. And then in a matter of seconds we should be able to um, see the CSV that I wanted. And so, we find that, you know, in a perfect world maybe all your data would be directly in Databricks, but in reality nobody has all their data directly in Databricks. And so, we wanted to make it really easy to bring

10:40in local files into this experience that you can combine that with your Databricks that's or your data that's on Databricks. Okay, awesome. Now that we have our data in I I guess I can give a just a quick brief overview of what we want to do. So, this is data on different hotels. So, we have the reviews, we have the cities, we have uh, information on the properties and the booking information like how expensive things were.

11:03And what I want to do is I want to first clean this data and then I'm going to join it together to make a a big table and then I want to do some aggregations on it. Because ultimately my boss said, "Hey, can you give me the data in this format?" So, I want to have the city where I Barcelona, I want to have an average sentiment, an average response time, and an overall rating, and an average total cost. So, that's my goal to get to this today. Um, and so,

11:27let's go and transform our data. So, the first thing that I maybe want to do here is I want to make um city tier an integer. So, right now it's a it's a string, tier one, tier two, for example, but this isn't super helpful for me as if I'm doing aggregations, I'd rather just have this be a number, one, two, three. So, I can go to Genie code and just say make tier an integer.

11:50Right? Really simple. Using natural language, I didn't have to dig around in any UI anywhere. I was just able to ask a question and then get an answer. And then Genie code's uh everything it does is agentic. Um and so it went and it just created uh this new operator for me where it casted it to uh an integer. The cool thing about Lexal Designer is

12:12that we show you the input and the output side by side so they can really easy validate what happened. So, for example, I can go to this row one, I see we have city ID one, Barcelona, coastal, tier one. Looks like coastal and then one and everything looks correctly if I look down here. Lake Como, we have three, Lake Como, tier three. Right? So, I can really easy validate that this is in fact the response I want and hit accept. Okay. You're like, "Okay, Jason,

12:34that's an easy one. That's not too bad. Let's do something a bit more complicated." I agree. Let's do something a bit more complicated. Um if I go over now to the properties, I have this amenities column and you can see it's like there's no spaces between these things and generally this is not very useful and in its current form. Um so, what I want to do actually is I want to take each amenity. So, we have like AC, kitchen, parking, pet friendly, etc.

12:57I want to make these their own column and then I want to have a true or false value um depending on if the property has that amenity. So, this is going to be pretty complex if I was to do this um you know, with with code or or doing the point and click way, but I can just again ask Genie code, I can say, um "Hey, um go uh through the amenities

13:20column and break each amenity into its own column with a true false value if that property has the amenity. Um so things like for example, uh BC true, uh kitchen false, etc.

13:43And so then Genie code's going to go away and uh as I mentioned, everything's agentic, so it might have to uh it might do a few little um side explorations to find out how to do this um and then should come back with uh an answer for us here. Um do you think you could do this uh Josip if you had to uh had to do this by hand? Do you think you'd be able to do it? >> Well, to be honest,

14:06um I became, let's say, so lazy and uh I know that the AI will be faster than me. So, I'll basically rely only on Genie code to do those kind of operation. And because everything is tailored uh like we have the meta data, we have all the info, the LLM can only

14:28be uh let's say, more powerful. And uh yeah, I mean, if it can be done in 2 minutes, why should I code it? I mean, I I I would only review, I can only review what has been written to make sure it's all it's perfect or it's good, and that's it. >> Exactly. Exactly. No, I I agree. You

14:50might be able to do it, but it's much faster to do it uh with AI, so it did it in about 20 seconds. Uh you know, I right when I asked you the question, it already had the answer. Um we can see we have a summary of the changes, what it did, and if we look at the output here, we can see that it looks like it got everything uh correctly. So, now I'm going to um accept this. All right. Now, the next thing I want to do is I want to join all these tables together. So, again, I could click

15:12through here and I could say, "Okay, let me create a join node and, you know, drag things and and connect it." But again, it's going to be way faster if I just say uh I with bookings, cities, hosts. Like I'm even going to leave the typo in there just to show um hosts and reviews. Um and I'll say don't keep duplicate columns as they are all the same.

15:35>> Oh, that that's one was painful. I remember when I used to join the tables together and sometimes we have the same names and then what I run as like, "Oh, you have a duplicate column." Then you remove the first one and then hey, yeah, you have a second duplicate column. And then you keep repeating the process for like maybe 15, 16 times until you get rid of all those similar columns. So

15:57either rename them, which doesn't make sense because technically it's the same name so have the same value. And it takes it takes a very long time. I remember it was so painful. >> Yes, yeah. So here we go. We have the answer now and so it looks like it joined with bookings first and then cities and then hosts um and then reviews here uh which which is great, right? And if I wanted to check the intermediate step data in

16:19each step, right? I could do that. Um but I'll just go over to this final one and see it looks like we have all the columns we have. Amenities joined looks like everything's good. If I wanted to, I could use the column explore here to kind of scroll through and make sure I have everything that I want. Um but generally um it looks like um everything is good here. So I will go ahead and uh accept that. Now I have one more thing that I want to enrich the data with before we

16:41do our aggregations and that is going to be analyzing the sentiments of the comment column. Um so we mentioned you know my >> boss wanted me to get the average sentiment. So first of all, I need to analyze the sentiment and to do that I can use an AI function. And one of the cool things is in Lake Flow Designer we have AI functions built directly into it. So previously they were very accessible. You had to use SQL but now you can just go into click to

17:04use your AI function. So I'll say I want to analyze the sentiments. We'll call this sentiment and then I want to analyze the comment column and hit apply here. And it's pretty amazing that in a matter of seconds we're going to have the sentiment for every single comment or something that you know only a few years ago would have taken a a long time to do. You have to import a custom library, run some Python code. Now

17:27with point and click you can just analyze uh the sentiment and and and apply that. So, there we go. We have that. Okay. >> No, you know that's what's with what you're doing. That was my basically my uh the project that took me 3 months when I was working for for a bank a few years ago. >> And here I can literally see the pipeline that you have built

17:51for in like I don't know 5 minutes? >> Yes. Yes. Yeah, exactly. It's uh I've been doing some cool stuff, but I think this next part is my favorite part of the demo and it's a pretty cool part. So, now we have the big table. Now we need to aggregate it and we need to make it into the format. So, let's go again and review this. So, we want to take a city, average sentiment, average response time, average total cost, and then make it fit this format. I have

18:14like some decimal places. So, there's two decimal places here. I only want one decimal place here. So, I have a pretty specific format. Now, I could go through and do this all by hand, but one of the cool things that we can do with like Photo Designer is the idea of transforming by an example. So, I'm going to take a screenshot of this here. I'm going to go back over here. I'm going to hit this image icon. And then I'm going to upload

18:36the image here so that we have it. And I'm just going to say make my data fit the format of this image. Now, all this is doing is I want to talk about just how complicated this is in the first place. So, the first thing is that we asked to do is average sentiment. Well, how do you have average sentiment when this sentiment column is a string? So, implicitly it's going to have to convert this into some numeric value that represents the sentiment. So,

19:00that's like the first step that it's probably going to have to do. If I go over here, then I didn't talk about rounding anything. I just gave it a screenshot. And so, I think we're going to find that it is going to be very smart and know how to um round these things, right? It's going to find the details that that we had. It's then going to do a bunch of different aggregations and then give us final answer. So, again, like if I was to do this by hand,

19:22it it would take some time, right? It'd be possible and I could do it, but it would take some time. But if we go back over here, we can pretty much see that the first thing it did was it says, "Oh, hey, I want to do sentiment numeric." And it looks like it did this. And what I can do here now is I can I can see what it did. It generates a a plain English description. Says, "Hey, I convert the sentiment text into a numeric score, one

19:45for positive, five for mixed, and zero for negative, and null for over any other value." That's like perfect. That's exactly what I wanted. The cool thing is if I wanted to tweak that, I can actually change the English here. So, maybe I want negative one for for for negative. I can just change this to be negative one. It changes the code and then it will re-render um for for for my my operator, right? So,

20:08that's that's that's awesome. I can just hit apply to change that. Um then it went over and it did an aggregation and this will take a second to show. And then let's see over here. And then it went and formatted the report. Um So, let's see. Okay, here's the aggregation now. So, we can see it says, "Okay, now I know how to do the aggregation. I have all the information.

20:31Let me run this." And so, we see it didn't have anything rounded here. And then if I click over to the format, it looks like it did all the rounding. So, I have the city, I have sentiment with two decimals, I have the average response time with one, overall rating with three decimals, and then have the average cost rounded to the nearest whole number. Um and so, we can see it it did it it did it for us, which is which is super awesome. And again, would have taken me

20:53a lot longer if I didn't just give it the screenshot example of how I wanted my my data um to look. So, I don't know you guys. I think that's pretty cool. What do you think? >> I I lost words, to be honest. It's It's I think it's way easier because it's I can build the flow step-by-step compared to using

21:15Genie code where you got to give the entire prompt and then keep adding. You can build the pipeline step-by-step to make sure everything is perfectly fine. Uh and make sure that the next steps are also working well. So, it's like in building brick by brick. So, this is for me it's perfect tool for for the for the analyst but also data engineers

21:38that just want to go faster. You can build your pipeline or your transformation in brick by brick and then you're sure every step is correct to move to the next one because it relies on the previous step and so on until you have a fully ready pipeline and I think the things you didn't show them that if I'm not wrong behind the scene it's it did the very it's only sequel.

22:01The two >> Yeah, yeah. It's It's actually It's actually PySpark now. We So, we recently made a change to make it from SQL to PySpark just because it gave us more powerful capabilities. Um but it is like code behind the scenes, right? Which is the which is the very important thing. So, like the cool thing here is that if I wanted to take this and I wanted to put this in Git, for example, I could just, you know, click this this Git modal and then I have basically a

22:23PySpark notebook behind the scenes um that I can I can store. I can put up in CI/CD um and I can I can do all of this stuff. Um so, it's it's backed by code. It's directly in Databricks. So, if I want to schedule this, right, I can add a schedule really easily. If I want to add this as a task to another production pipeline or job, I can do that really really easily. So, taking this what we just built it behind the scenes, it's all performed PySpark code, which

22:46is uh which is really great. So, I think yeah, that's that's pretty much what I wanted to show you all today. Um and yeah, we're excited to see what you build with Lake Formation Designer and let us know if you have feedback. We're obviously still pretty early in product development, so there's still some things we're perfecting. Um and if you have any ideas of how we can make it even more useful for for you, we would we'd love to hear.

23:09>> And I guess and I if I'm not wrong I guess that Lake for designers going to be soon available on the Databricks free edition. >> Yes, it should be available on free edition in a matter of days actually. So, by the time you all see this video, it will probably be available on free edition. >> So, go try it easily. Build your own

23:33pipeline, grab data from Kaggle or maybe your data you have on your CSVs or whatever format. Drag and drop them like Jason shows you. Build your pipeline and then use Genie to go and query your data. Give it a try. Trust me, you're going to be amazed. This is your way and then if you have any questions you can reach out

23:56to Jason Maser on LinkedIn. >> Awesome. Awesome. >> Thank you. Thank you so much Jason and uh maybe we're going to see you for the fourth time fourth time in few weeks. >> Yeah, I would love to see you. I'd love to come back and show you all the awesome features we have in the pipeline. There's a lot coming, so. >> I have no doubt. Bye buddy. >> In Lake Cow, Lake data seas, Lake flow

24:20streams to flow. Iceberg and Delta Lake, tables a glow. Unity Catalog governs Lake base, OLTAP strong. Genie takes to SQL, MOSAIC AND KO PROLONG. You sipping querying cohorts like the fire. Data and dream scale higher and higher. Next gen, drop it.