LakeFusion
Introduction to MDM
Hey, today I have the pleasure to host Ros King, chief architect. Hi, Ros. Hey, Yousef, thank you for having me.
Appreciate it. Can you introduce yourself? Tell us more about you and the company you're working for and then we can introduce today's subject. Sure, can do. So, my name is Roz King. I work at
Lake Fusion as the chief architect. I kind of hate backgrounds. It it just feels like bragging when we just go and list these long strings of things we've done. I've been a data engineer for for
most of my career and have bounced around from from enterprise to academia to nonprofit. So I've I've seen a little bit of everything and made every single mistake in the book. And so I'm really
excited to spend my time right now working on on master data management the problems there because it's a common problem you see across uh a lot of enterprises. And I think you mentioned a
very interesting keyword master data management aka MDM. So can you tell us more about the MDM and what's the purpose of using an MDM or master data management? So I I think this is really
interesting because if you ask someone what MDM is in the industry who's in the know 10 people you get 11 different answers. There's there's a lot there and the landscape of MDM is huge. So
generally we think about what MDM or master data management is is a process of mastering master data. So there's a lot to unpack there. Master data it's it's useful to define that. Master data
is essentially the data about your business and about your business entities that your entire business refers to and gives context to data data transactions. So you can think about
this as customers or um suppliers, products, all of the core entities that your business runs on and the information about them that your business needs to operate correctly.
Does that make sense? Yeah. Yeah. Very very clear. It's like understanding the the context behind the data you do have and make sure that those users may find the data easily and understanding the
context behind every single assets you have on the on your system. Yeah. So that's the master data. And the thing is with a lot of modern enterprises, we have these huge sprawling enterprises
that have multiple data sources and multiple different types of data quality, multiple definitions of the same thing. And so the process of actually mastering that master data is really tricky. It involves the IT side
who has to go in and process all of that data, implement business rules, but it also involves the business side that has to define those business rules, has to own those business rules to a degree on
a collective definition of those rules across the entire business. When you have maybe a bunch of different use cases for those rules, you have regular regulatory compliance. It's a truly
complex field. And so master data management is essentially the process of going from all of the raw data and turning it into master data that the business can action on and trust and
rely on actually run their business correctly without all the issues that come up when you have these scattered data sources across their entire system.
And just for the clarity of maybe people who never heard about MDM. So when you're saying master data management is it maybe a SAS application that will come that will that you will have to
connect with your with your I don't know data lake or or the storage where we have all your data or any system and then is this mapping or let's say the discover of the data done automatically
or you still need the business team to do all the work or is it a combination between the business team and the maybe the intelligence inside an MDM that's a good question and I I think again the
answer to that differs depending on who you're talking to. Now the core problem of master data management is getting the business actually into the data and working with the data. And so you know
for a good example if you're running on data bricks and you have a really strong engineering team that's processing a lot of data you still need to somehow expose that to the business so the business can
interact with it run their stewardship on that data. And you know, Unity catalog as a governance tool is a really great example of how you start bringing in some of the business logic into the
system, but you still have this gap is how do you actually let the engine or the the business team interact with the data itself? And that's where your statement of is this a is this a SAS
tool or or what what actually is this that makes this happen? It's generally a collection of tools because there's a lot of different things that go on in the mastering data pipeline. There's
there's a lot of different steps in master data management, but one of the core pieces of tooling that's necessary is some way of exposing that data to the business users so they can interact with
it and allowing them to interact with all the processes, the data cleaning, the data profiling, the entity resolution, all of those pieces, the defining of rules that that you need in
order to understand how to how to resolve these data entities, how to apply survivorship to the entities, all of those tools need to be exposed in a way that a business user can interact
with them. So in short, yes, this this is a tool. This is a collection of tools. It is a bunch of processes you have to build around these tools to make sure that the business can get into the
data and work with the data and see it and understand the implications of the actions that they're taking on. Yeah, it's literally building uh the entire knowledge base of the business people
inside the platform to make sure that you have business continuity and you have a documentation inside the platform to make sure that if I am maybe I don't know new joiner I understand how this data has been
computed what's the I don't know system using this and like and why this I don't know this data is important yeah yeah that's exactly right and if I'm not wrong for Lake Fusion has been built on
the top of data bricks. That's right.
Yes, that is right. We've been talking about MDM just in general, but Lake Fusion, the company I work for, we have built an MDM, an end toend MDM tool that sits on data bricks. Now, this is really
cool for a couple different reasons, but the main reason that it's really exciting for folks who are already on data bricks is a lot of the traditional MDM tools basically had to build their
own data platform just to run MDM. Which means if you subscribe to any of the existing MDM software as a service tools, what you end up having to do is pull your data out of your current data
ecosystem into theirs, run all your mastering processes there, and then push them back into your data ecosystem. So then you can actually use that data. And part of that is is is because, you know,
prior to cloud, everything was running on prem. You had to deploy some very specific systems. There's a lot of different ways to build these integrations. And so a lot of these entrenched players came up through the
on-prem and then now trying to adapt to cloud. Now that we're, you know, deep into not just cloud, but into the lakehouse environment, it's time to rethink actually how we architect and
build our MDM systems. And so that's why at Lake Fusion, we started thinking about what would a modern version of an MDM system look like. We immediately said, well, this sits right in data
bricks. Your data should never leave data bricks. Data gravity, I think, is the term that we throw around a lot.
like you you want to keep your data like bring the compute to the data and so that's what we're doing with Lake Fusion. We we're we're saying you know data bricks has this incredible AI
platform that built exposes all the tools we need to take data from across a bunch of different sources try to figure out how they match how they merge how to dduplicate them and expose all of that
to the business users. So Lake Fusion builds this really nice user interface on top of data bricks that allows business users to come in and interact with the incredible power that data
bricks offers. Yeah. So to understand you you're bringing the zero copy data and you use the customer's compute. Yes.
Then you're so behind the scene how does it work? Is it like a deployment using I don't know data bricks assets bundles it scripts or that you're sharing with a customer. So this customer can deploy
the platform on the top of data bricks or it's um how how does it work? Good question. So the way a customer gets started is we they install us from any of the large cloud marketplaces. So AWS
marketplace, Azure marketplace. We're not on GCP right now, but we will be fairly soon. And what that does is it deploys a Kubernetes cluster into the customer's cloud environment. that Kub
Kubernetes customer connects to the datab bricks um compute layer using the OOTH SDK. The user needs to be on Unity catalog because we use all the Unity catalog information to pull in the
user's data. So that means when a user connects Lake Fusion to their own Unity catalog instance and starts browsing for the data they want to actually master, they end up just pulling up the list of
all their data assets in Unity catalog.
The process of importing data into lake fusion is actually just registering the location of that asset in Unity catalog.
All compute happens through notebooks or it happens through the uh data bricks SDK which means all the compute is happening on the existing clusters using serverless clusters. there's no
additional infrastructure to spin up on top of the really lightweight user interface like Kubernetes clusters that actually run our UI. So basically for the requirement you just need to have of
course a datab bricks data bricks account sign up through the marketplace marketplace Asia marketplace and coming soon the GCP marketplace and then um it's like um click point click and then
you're good to go no need you know don't need to set up any permissions everything is done is done for you just next next and you're good to go. Yep, that's exactly right. Awesome. And for
example, is there any trial period if someone want to try Fusion or how does it work? Good question. So, right now we don't have a trial period. We did have a readonly interface that we can give
people access to to just click around to get a better understanding of what things look like, but but we were finding that the time to value from using Lake Fusion is shortened so significantly over a lot of the
entrenched MDMs that we want to trick quickly get to just doing a pilot in our customers environments. That means when we start having sales calls and talking to customers and they say, "Hey, I'm
interested. I want to try this out. I want to see what this can do for us." We just say, "Great. Let's just install it in your environment. Let's get to work.
Let's run a really quick 6 week to 8 week pilot on real data, deploy this data into production, and just start get this going." And this is another really cool thing about Lake Fusion. A lot of
traditional MDMs can take 6 months, 8 months to just get to a final mastered entity. We turn this around in 6 weeks to 8 weeks. It is a wildly fast time to value. And a lot of that is because
there's no data integrations that have to happen. We use a really advanced AI engine that doesn't need the user to write a bunch of match merge rules, which is another huge part of the of the
process in building out MDM. And so we can we can cut down the time to value really quickly, which means we stop having conversations about can we do a free trial and start having conversation
about how can we deploy into prod in the matter of a couple weeks. That that's very interesting. It means you go straight to the point. You you show the the real value of the platform because
sometimes like I don't know 14 days trial or one month trial can cannot be enough or maybe they need additional resources while your team is here to help them get started quickly and
benefit from the from the from the platform. So if someone want to reach out like for I don't know to your team is there any like uh link or email or person to reach out to? Yes, absolutely.
So, on our website, we have a contact form, lakefusion.ai. You can come reach out to us and we will schedule a meeting. We can do demos. We have a demo on YouTube that we can share. So, anyone
who wants to see kind of the look and feel of the platform can watch that demo and it goes does a pretty deep dive into all the features that Lake Fusion offers. Maybe if you can you show us
like just the interface of the lake Fusion. Yes. Yes. Give me a second and pull it up. So, this is the general user interface for Lake Fusion. You know, I mentioned earlier that master data
management is a process. It's a process that involves data profiling. It involves executing data quality operations. It involves defining your entities, involves building match models. What you see in the Lake Fusion
UI here is essentially that process laid out into a section of modules. So, we have the ability to import data into the Lake Fusion tool. This process of importing data essentially brings up
your Unity catalog and you can browse through that Unity catalog and select data that you want to operate on. The next step in the process is data profiling. This allows you to understand
the overall quality of your data. Look for missing values. Look for frequency distributions etc. of your data. Operate and run data quality notebooks. This can attach to existing um data bricks notebooks or we
have this really nice visual programming tool that allows users to build actually like you know noode data quality operations. What's really cool is this gets then compiled down to a data bricks
notebook on the back end which means we can now do you know noode data processing at scale for business users which ends up being a really key feature for teams that want to give their data
stewards their business users end toend control over a master data management program but still be able to use the power of data bricks under behind the scenes. Yeah, this is amazing because
since it's low code, no code like you don't require a technical person to use this like can just bring the as you mentioned the uh business folks and they can start implementing the rules they
want to have maybe for data quality or maybe for entity resolution and bring their intelligence inside your uh diffusion. Yep, that's exactly right. I will I will not go deep into these other
pieces just because you know I'll we'll we'll share that YouTube link and let other people's go people go take a look.
The one thing I do want to show which I think is really cool is the entity search. This is the piece that I think helps people understand really what what this ends up looking like. So this is a
stream where Stuart will go to actually try to find some potential matches. The way our our AI matching works is we essentially give it two records and ask an AI model, hey do you think these are
the same records? and it uses the entire context of that record. We pass it and it uses the business context of of that record. And what this is really cool because it means the user doesn't have
to define any matching merge rules. Our AI really just looks at the full record, which means if the vendor name is different, it's able to still find that match. If the vendor region is
different, it's able to still understand the match because it understands semantically what these things mean. And it allows us to find matches in data that a lot of the entrenched systems
just cannot match. Yeah. And this is very very important because I remember I used to work for a company and we had an entire project just to make sure that we have the the right naming because we had
to do an aggregation to see the how many products we bought from this specific supplier and since they had different names it was hard for them to do the right aggregation. So with the the right
naming you will have better aggregation which means let's say a more uh like big number of this product. So we can go to this company say okay I bought 1 million units from this specific product. So
give us a return of 10% because of this and without this right entity you would end up having less units and which means low lower discounts from this specific company. Yeah, that's a really that's a
really good example. And I think this this kind of showcases why kind of rethinking MDM in in kind of this AI world makes a lot of sense. Like the kind of matches you're talking about,
like you can you can get there, you can get close when using some fuzzy matching and similarity searching, but because we're using vector search and we're using LLMs to do this, this means we're
actually looking semantic similarity rather than just like how how close is the string to another string, for example. and and it means that we're able to find matches that are semantically similar and there's there's
not really anyone doing MDM. Yeah, it's very very interesting. It seems to be a very good product. Thank you so much, Rose. And u see you maybe very very soon maybe for a release of an update on Lake
Fusion. Can't wait. Thank you for having me. I appreciate it. Bye. See you.