LakeFusion MDM on Databricks: Enterprise Data Unification

0:05Hey, today I have the pleasure to host Ros King, chief architect. Hi, Ros. Hey, Yousef, thank you for having me. Appreciate it. Can you introduce yourself? Tell us more about you and the company you're working for and then we can introduce today's subject. Sure, can do. So, my name is Roz King. I work at Lake Fusion as the chief architect. I kind of hate backgrounds. It it just

0:28feels like bragging when we just go and list these long strings of things we've done. I've been a data engineer for for most of my career and have bounced around from from enterprise to academia to nonprofit. So I've I've seen a little bit of everything and made every single mistake in the book. And so I'm really excited to spend my time right now working on on master data management the problems there because it's a common problem you see across uh a lot of

0:51enterprises. And I think you mentioned a very interesting keyword master data management aka MDM. So can you tell us more about the MDM and what's the purpose of using an MDM or master data management? So I I think this is really interesting because if you ask someone what MDM is in the industry who's in the know 10 people you get 11 different

1:13answers. There's there's a lot there and the landscape of MDM is huge. So generally we think about what MDM or master data management is a process of mastering master data. So there's a lot to unpack there. Master data it's it's useful to define that. Master data is essentially the data about your business and about your business entities that your entire business

1:37refers to and gives context to data data transactions. So you can think about this as customers or um suppliers, products, all of the core entities that your business runs on and the information about them that your business needs to operate correctly. Does that make sense? Yeah. Yeah. Very very clear. It's like understanding the context behind the data you do have

1:59and make sure that those users may find the data easily and understanding the context behind every single assets you have on the on your system. Yeah. So that's the master data. And the thing is with a lot of modern enterprises, we have these huge sprawling enterprises that have multiple data sources and multiple different types of data quality, multiple definitions of the

2:21same thing. And so the process of actually mastering that master data is really tricky. It involves the IT side who has to go in and process all of that data, implement business rules, but it also involves the business side that has to define those business rules, has to own those business rules to a degree on a collective definition of those rules across the entire business. When you

2:43have maybe a bunch of different use cases for those rules, you have regular regulatory compliance. It's a truly complex field. And so master data management is essentially the process of going from all of the raw data and turning it into master data that the business can action on and trust and rely on actually run their business correctly without all the issues that

3:05come up when you have these scattered data sources across their entire system. And just for the clarity of maybe people who never heard about MDM. So when you're saying master data management is it maybe a SAS application that will come that will that you will have to connect with your with your I don't know data lake or or the storage where we have all your data or any system and then is this mapping or let's say the

3:27discover of the data done automatically or you still need the business team to do all the work or is it a combination between the business team and the maybe the intelligence inside an MDM that's a good question and I I think again the answer to that differs depending on who you're talking to. Now the core problem of master data management is getting the

3:49business actually into the data and working with the data. And so you know for a good example if you're running on Databricks and you have a really strong engineering team that's processing a lot of data you still need to somehow expose that to the business so the business can interact with it run their stewardship on that data. And you know, Unity catalog as a governance tool is a really great example of how you start bringing in some of the business logic into the

4:12system, but you still have this gap is how do you actually let the engine or the business team interact with the data itself? And that's where your statement of is this a is this a SAS tool or or what what actually is this that makes this happen? It's generally a collection of tools because there's a lot of different things that go on in the mastering data pipeline. There's there's a lot of different steps in master data management, but one of the

4:34core pieces of tooling that's necessary is some way of exposing that data to the business users so they can interact with it and allowing them to interact with all the processes, the data cleaning, the data profiling, the entity resolution, all of those pieces, the defining of rules that that you need in order to understand how to how to resolve these data entities, how to apply survivorship to the entities, all

4:56of those tools need to be exposed in a way that a business user can interact with them. So in short, yes, this this is a tool. This is a collection of tools. It is a bunch of processes you have to build around these tools to make sure that the business can get into the data and work with the data and see it and understand the implications of the actions that they're taking on. Yeah, it's literally building uh the entire knowledge base of the business people

5:20inside the platform to make sure that you have business continuity and you have a documentation inside the platform to make sure that if I am maybe I don't know new joiner I understand how this data has been computed what's the I don't know system using this and like and why this I don't know this data is important yeah yeah that's exactly right and if I'm not

5:42wrong for Lake Fusion has been built on the top of Databricks. That's right. Yes, that is right. We've been talking about MDM just in general, but Lake Fusion, the company I work for, we have built an MDM, an end toend MDM tool that sits on Databricks. Now, this is really cool for a couple different reasons, but the main reason that it's really exciting for folks who are already on

6:04Databricks is a lot of the traditional MDM tools basically had to build their own data platform just to run MDM. Which means if you subscribe to any of the existing MDM software as a service tools, what you end up having to do is pull your data out of your current data ecosystem into theirs, run all your mastering processes there, and then push them back into your data ecosystem. So

6:27then you can actually use that data. And part of that is is because, you know, prior to cloud, everything was running on prem. You had to deploy some very specific systems. There's a lot of different ways to build these integrations. And so a lot of these entrenched players came up through the on-prem and then now trying to adapt to cloud. Now that we're, you know, deep into not just cloud, but into the lakehouse environment, it's time to

6:50rethink actually how we architect and build our MDM systems. And so that's why at Lake Fusion, we started thinking about what would a modern version of an MDM system look like. We immediately said, well, this sits right in Databricks. Your data should never leave Databricks. Data gravity, I think, is the term that we throw around a lot. like you you want to keep your data like bring the compute to the data and so that's what we're doing with Lake

7:12Fusion. we're we're saying you know Databricks has this incredible AI platform that built exposes all the tools we need to take data from across a bunch of different sources try to figure out how they match how they merge how to dduplicate them and expose all of that to the business users. So Lake Fusion builds this really nice user interface on top of Databricks that allows business users to come in and interact

7:35with the incredible power that Databricks offers. Yeah. So to understand you you're bringing the zero copy data and you use the customer's compute. Yes. Then you're so behind the scene how does it work? Is it like a deployment using I don't know Databricks assets bundles it scripts or that you're sharing with a customer. So this customer can deploy

7:57the platform on the top of Databricks or it's um how how does it work? Good question. So the way a customer gets started is we they install us from any of the large cloud marketplaces. So AWS marketplace, Azure marketplace. We're not on GCP right now, but we will be fairly soon. And what that does is it deploys a Kubernetes cluster into the customer's cloud environment. that Kub

8:20Kubernetes customer connects to the datab bricks um compute layer using the OOTH SDK. The user needs to be on Unity catalog because we use all the Unity catalog information to pull in the user's data. So that means when a user connects Lake Fusion to their own Unity catalog instance and starts browsing for the data they want to actually master, they end up just pulling up the list of all their data assets in Unity catalog.

8:43The process of importing data into lake fusion is actually just registering the location of that asset in Unity catalog. All compute happens through notebooks or it happens through the uh Databricks SDK which means all the compute is happening on the existing clusters using serverless clusters. there's no additional infrastructure to spin up on top of the really lightweight user interface like Kubernetes clusters that actually run our UI. So basically for

9:07the requirement you just need to have of course a datab bricks Databricks account sign up through the marketplace marketplace Asia marketplace and coming soon the GCP marketplace and then um it's like um click point click and then you're good to go no need you know don't need to set up any permissions everything is done is done for you just next next and you're good to go. Yep,

9:31that's exactly right. Awesome. And for example, is there any trial period if someone want to try Fusion or how does it work? Good question. So, right now we don't have a trial period. We did have a readonly interface that we can give people access to just click around to get a better understanding of what things look like, but but we were finding that the time to value from using Lake Fusion is shortened so significantly over a lot of the

9:53entrenched MDMs that we want to trick quickly get to just doing a pilot in our customers environments. That means when we start having sales calls and talking to customers and they say, "Hey, I'm interested. I want to try this out. I want to see what this can do for us." We just say, "Great. Let's just install it in your environment. Let's get to work. Let's run a really quick 6 week to 8 week pilot on real data, deploy this data into production, and just start get

10:16this going." And this is another really cool thing about Lake Fusion. A lot of traditional MDMs can take 6 months, 8 months to just get to a final mastered entity. We turn this around in 6 weeks to 8 weeks. It is a wildly fast time to value. And a lot of that is because there's no data integrations that have to happen. We use a really advanced AI engine that doesn't need the user to

10:38write a bunch of match merge rules, which is another huge part of the of the process in building out MDM. And so we can we can cut down the time to value really quickly, which means we stop having conversations about can we do a free trial and start having conversation about how can we deploy into prod in the matter of a couple weeks. That that's very interesting. It means you go straight to the point. You you show the real value of the platform because

11:01sometimes like I don't know 14 days trial or one month trial can cannot be enough or maybe they need additional resources while your team is here to help them get started quickly and benefit from the from the from the platform. So if someone want to reach out like for I don't know to your team is there any like uh link or email or person to reach out to? Yes, absolutely.

11:25So, on our website, we have a contact form, lakefusion.ai. You can come reach out to us and we will schedule a meeting. We can do demos. We have a demo on YouTube that we can share. So, anyone who wants to see kind of the look and feel of the platform can watch that demo and it goes does a pretty deep dive into all the features that Lake Fusion offers. Maybe if you can you show us like just the interface of the lake Fusion. Yes. Yes. Give me a second and

11:47pull it up. So, this is the general user interface for Lake Fusion. You know, I mentioned earlier that master data management is a process. It's a process that involves data profiling. It involves executing data quality operations. It involves defining your entities, involves building match models. What you see in the Lake Fusion UI here is essentially that process laid

12:11out into a section of modules. So, we have the ability to import data into the Lake Fusion tool. This process of importing data essentially brings up your Unity catalog and you can browse through that Unity catalog and select data that you want to operate on. The next step in the process is data profiling. This allows you to understand the overall quality of your data. Look for missing

12:33values. Look for frequency distributions etc. of your data. Operate and run data quality notebooks. This can attach to existing um Databricks notebooks or we have this really nice visual programming tool that allows users to build actually like you know noode data quality operations. What's really cool is this gets then compiled down to a Databricks

12:55notebook on the back end which means we can now do you know noode data processing at scale for business users which ends up being a really key feature for teams that want to give their data stewards their business users end toend control over a master data management program but still be able to use the power of Databricks under behind the scenes. Yeah, this is amazing because

13:18since it's low code, no code like you don't require a technical person to use this like can just bring the as you mentioned the uh business folks and they can start implementing the rules they want to have maybe for data quality or maybe for entity resolution and bring their intelligence inside your uh diffusion. Yep, that's exactly right. I

13:42will I will not go deep into these other pieces just because you know I'll we'll we'll share that YouTube link and let other people's go people go take a look. The one thing I do want to show which I think is really cool is the entity search. This is the piece that I think helps people understand really what what this ends up looking like. So this is a stream where Stuart will go to actually try to find some potential matches. The way our our AI matching works is we

14:05essentially give it two records and ask an AI model, hey do you think these are the same records? and it uses the entire context of that record. We pass it and it uses the business context of that record. And what this is really cool because it means the user doesn't have to define any matching merge rules. Our AI really just looks at the full record, which means if the vendor name is

14:27different, it's able to still find that match. If the vendor region is different, it's able to still understand the match because it understands semantically what these things mean. And it allows us to find matches in data that a lot of the entrenched systems just cannot match. Yeah. And this is very very important because I remember I used to work for a company and we had an entire project just to make sure that we have the right naming because we had

14:52to do an aggregation to see the how many products we bought from this specific supplier and since they had different names it was hard for them to do the right aggregation. So with the right naming you will have better aggregation which means let's say a more uh like big number of this product. So we can go to this company say okay I bought 1 million

15:16units from this specific product. So give us a return of 10% because of this and without this right entity you would end up having less units and which means low lower discounts from this specific company. Yeah, that's a really that's a really good example. And I think this this kind of showcases why kind of rethinking MDM in in kind of this AI

15:40world makes a lot of sense. Like the kind of matches you're talking about, like you can you can get there, you can get close when using some fuzzy matching and similarity searching, but because we're using vector search and we're using LLMs to do this, this means we're actually looking semantic similarity rather than just like how how close is the string to another string, for example. and and it means that we're able to find matches that are

16:02semantically similar and there's there's not really anyone doing MDM. Yeah, it's very very interesting. It seems to be a very good product. Thank you so much, Rose. And u see you maybe very very soon maybe for a release of an update on Lake Fusion. Can't wait. Thank you for having me. I appreciate it. Bye. See you.