CDP + Delta Sharing: Amperity & Databricks Integration

0:00and every single time we ran it it cost $1,800 and took 6 hours it gets pretty expensive right and I because I feel like people think okay like it's kind of easy I can just match the email and it's going to be enough it's going to work but like it's much more than that we've got the only proven solution that handles machine learning for that and so we have 45 different models that will match customers based on the available data that they have across all those different

0:24channels hey folks so today uh I'm with youf and calb calb is working for and we will be talking about CDP today hey Caleb thanks for joining um maybe do you want to start presenting yourself yeah my name is Caleb benningfield I'm the head of Lake housee strategy and imp parody uh I kind of uh we've been around since 2016 and I've been there pretty

0:46much the whole time uh I've been doing startups for quite a while and uh uh we've uh you know I started off doing uh software development uh work for us and Client Services work and then uh eventually you know kind of migrated into a bit of a uh you know startup within a startup model and did a lot of sort of research and development uh projects and recently I've been

1:09spearheading our Lakehouse initiative building out U um connectivity with Databricks specifically and uh I'm excited to talk more about it um but yeah I've been doing startup for quite a long time and uh looking forward to uh to talking about the cool stuff we've built we've seen during dice uh during the data ni Summit our annual conference mate was on the stage and he introduced amperity as a new

1:34partner on the Delta sharing side if I'm not wrong and also on the CDP so can you tell us more about the partnership you do have and more and also tell us more about the company you're working for yeah I I mean so you know uh the we we are the first CDP to have a Delta sharing based interf or integration with

1:58uh Databricks uh basically the idea for us CDP is a very odd term you know there's a lot of different companies that call themselves that and I and you know it's sort of a as a category not it lacks a very unified definition um we uh but what our our version of that premise is that uh you know everybody's got an incredibly large amount of data and uh um you know an Enterprise company will

2:21have hundreds or maybe thousands of tables that they have to manage and each of those will have valuable things and if they want to truly understand their customers uh you know they have to standardize and unify that data and we've built a tool that's really specializes to and to flexibly let people uh you know create a single really high quality data asset out of uh

2:44all of their their disper data sources and then make that available in a bunch of different places and so we connect using a Delta sharing uh uh connection to uh Databricks and then we use machine learning to generate an identity graph and then we have a marketer interface for the resulting data and so what you end up with when you use imperity is really high quality data Assets in Databricks for you to build

3:07AI on or for you to build your own analytics dashboards or custom models uh and but you also end up with an interface for your business teams to log in and deploy paid media campaigns or uh uh you know onboard Salesforce or adobe if that's what you want and so basically it's a it's a solution package that you know connects to your Databricks

3:29account and offers you a like really nice data modeling tool as well as something that it can be your profile manager and sort of onboarder for all of your different uh uh marketing channels all right so so CDP customer data platform I imagine you connect um like with marketing system with Erp maybe Salesforce can can you can you give us a few example you know which data you

3:52consume uh and maybe a few yeah a few customer using empar so that we have a better understanding yeah absolutely so um uh you know basically uh you know companies will get data from all sorts of places one of the most common ones will be an eCommerce platform so their online store so that has profiles and transactions but then they might have a loyalty program that has profiles they

4:15might have physical stores where you get transactions and interactions you might have uh you know web events uh and visits that you want to Wrangle uh you have um uh uh you know response data from your emails that that you've sent in the past that tell you lots of things about your about your your customers uh you want to know you know what products they've looked at online and offline and

4:37what their purchase behaviors like in the past and and you want to know uh uh you know also like what their preferences are so an increasing you know kind of push has been gdpr and CC CCPA for California and sort of data privacy practices so being able to understand you know your customer across all these touch points is actually not

4:59just about marketing anymore it's also about operations you have to be able to identify and and and manage data in that way too and so we you know we'll get compliance requests will get preferences and opt status and that kind of stuff and so uh you know all of that data is shaped in a very different way and sort of turning it into something useful is you know often like thousands of hours of work to write a bunch of code to do that and and we're trying to make that

5:22more accessible uh because you know if you're if you're one of our customers I'll give you some examples we have them all over the uh the kind of different verticals we like I personally implemented our first customer Alaska Airlines um and we have a couple of travel and Hospitality style customers uh we actually have pretty you know all of the Seattle sports teams except for the Sounder the storm so we've got

5:44Mariners and and Seahawks and uh in you know we have an NFL in in Seattle Kraken uh deal but we also have a bunch of a ton of retail customers like tapestry Dick Sporting Goods the Gap that kind of thing and then we also have like uh you know of cpg grocery customers your Nestle and your recit uh you know uh lblock type of companies and so like

6:07it's you know we have we have uh Enterprise companies from all these different verticals you need something that's flexible enough because each of those businesses is so different that the airline companies care about passengers and reservations but the retail companies care about uh uh you know uh you know online offline purchases more and like specific products and what part of the store they're interacting in and so you have to have something that can handle all

6:30those situations but uh but we and uh we built something that handle that Enterprise scale and kind of you know tailoring it to a company and and and I think one of the main challenges when you have all these systems and maybe different brand within your organization each system might have the same customer say for example with like an email but maybe it's not the same email or maybe there is a typo maybe we just have the

6:52phone number uh so one of the big value of CDP is that you can connect all these platforms and you know try to identify and potentially Coss sell on the same customer uh and I think you do things like that like to match the customers is that one of the main news case you have yeah that's actually your kind of primary differentiation we have uh uh you know identity resolution most of the time is handled by rules and so people

7:14will say if they're email address matches and their last name or and they might create a Cascade of those or they might use fuzzy matching but we've uh we've got the only proven solution that that handles machine learning for that and so we have 45 different models that will match C customers based on the available data that they have uh across all those different channels and so to your point like uh you know you'll have

7:37profiles and all those different systems and the uh the hard part is like the most active customers your most pass passionate customers generate the most data because they make the most purchases they interact with your brand the most and if you don't you know uh have a really nice identity resolution set up then you will end up treating them like two or three different people depending on how they interact which can mean you know risk but can also mean

8:00that you're not optimizing you know their experience yeah so entity for everybody entity resolution is the process of matching uh these personel together based on you know is it the same email close to the same email maybe using phone and and things like that like it's I know it's a really big challenge especially when you have a lot of data because it can quickly become a quadratic issue because you have to for

8:22every entry if you have 1 million you have to match every entry against the you know the full 1 million uh and I know it's really hard to do if you do it yourself uh and and I think CB you you somehow magically even do it at scale without any real issue is that can you tell us more and how it's maybe how it's working yeah so back in 2016 uh so I mean oh actually I'll step back even

8:44further so before amperity uh uh the founders of amperity were a founders of a company called aperture which was a competitor with exact target so it was like email marketing and Journeys type of tool um and uh uh our key learning when I was at that company too our key learning there was how much difficulty it was for people to just get data

9:08modeled and onboarded for Enterprise companies so that they could even send the marketing and and and do the personalization work and so we spent we set out here to try to solve that problem and we worked with University of Washington uh Professor Dan suu as well as uh we have our own phds uh yan yan and our our chief architect stepen miles

9:30and you know they all got together and basically took a academic paper about entity resolution and uh figured out how to turn that into an actual product collaboratively over the course of a couple of years and uh then we B you know figured out you know uh how to make that you know performant like so the very first uh uh time we ever did Stitch

9:52uh was 40 million profiles and every single time we ran it it cost $1,800 and took six hours like and that's like so if you want to run that once a day I mean that's pretty gets pretty expensive right and uh uh now that same Stitch after you know eight years is like 10 or 12 minutes and costs you know I don't know $40 50 like so there's been a lot

10:14in terms of refining the rules but also figuring out how to scale that and how to turn that into something that's you know flexible enough for people uh and so uh uh you know we now our big you know that 40 million Stitch that took six hours now our bigger stitches are in the billions so we have you know there's only 8 billion people on Earth so uh you know and most companies don't have access to everyone but they do have a

10:36lot of duplicate records and they have a lot of uh um um you know sort of you know if you do flights for example they often don't have a profile on the flight data everything's a reservation so you have to like extract people out of that and so uh uh basically we worked with you know academics adapted that into a proper product and then figured out how to turn that into something that was

10:58scalable and profit over the and you know for for about five or six years now that's been uh um you know one of the biggest uh you know kind of key differentiations for what we do yeah and and honestly um I mean I I did a bit of that in my previous job um so I just want to insist on how hard it is when you have so many customers you like you have to do crazy stuff you have to

11:21kind of create buckets because you can you can always you know compare entities you think are kind of similar so you have to do tons of pre-processing cleaning then you have to create some buckets and then you have to match the bucket together like it's really like it's super hard to implements the result are often you know not good uh and and and I because I feel like people think

11:43okay like it's kind of easy I can just match the email and it's going to be enough it's going to work but like it's much more than that well and I mean you got to think about you know uh it really is sort of this meat between you know digital and human right like people you know have nicknames and like one of the biggest ones that gets me is like most people have multiple email addresses but most people

12:06use one email for for like their shopping and one for work stuff right but like the thing that almost every rules-based algorithm loses is like you know half the people on Earth are women and like a very common social Norm would be for them to change their name or their last name when they get married and almost every rules based algorithm immediately treats that like an entirely new person and that's like and it's

12:28hilarious to me that everybody just sort of lives with that and so you know but we're able to look at it and say no but the email address hasn't changed and like we also see the that the address is is similar and we're able to like you know understand from their actual profile data but also from their interaction history that this is very likely the same person and then and then do that with a transparency that tells you exactly how confident we are about

12:51that and then you can decide you know how like whether or not you want to take that match or not but like the you know that's like like you know a simple example but that's the kind of thing that happens across uh data in the real world you know someone will be Kimberly and someone will be Kim and even like a lein distance algorithm that's too big of a difference most of the time that's too many characters different to match that so you need to be able to handle

13:13nicknames and how common a name is and you know there's all sorts of different angles to that and so that's why it takes 45 different models to kind of get to a really accurate result and also you have to remember that like some data is very comprehensive if you're buying online you have to give someone your name and your address and you're billing information and all of that uh if but if you buy something you know you know in a you know in a store you sign up for a

13:36newsletter you might they might not have almost anything about you and you have to have an algorithm that can handle both very populated data and also very sparse data because if you only have an email address then that might be good enough but if you have email names phone numbers all this other stuff then you might have you know you need to be more robust than that but you can't you shouldn't have to have three different jobs that should be able to figure it

13:58out and apply only the models that that are valuable for that data I I I think I think now I understand why Quinton wanted to be with us when it when I told him that we're going to record a a session with um amperity because he's like he used to work on this subject so it it does it it does

14:20make sense and I think you can you can do an example on him because he he has three like uh three emails with and trust me you cannot guess them like he has like an email with Q dates.com Quinton dates.com so maybe you need to test with algorithm against his email and see what would be the result it would be interesting yeah I mean and that's that's actually one of the models

14:43one of the 45 models will take an email address and we'll chop it up and we'll say okay so every email is handle domain. extension and like Kaleb benningfield would like very common patterns would be C benningfield or you know Caleb B or whatever you know and if and we will like compare the handle versus the other information that we have and and that you can sometimes get

15:05extra confidence from that but you're right like there's so many different patterns and so are you using kind of gen model so U because I could see maybe it's super basic but I could see how you can say Hey you know I have these two customers they look the same that's what I know do you think it's the same customer and you would send that to I don't know Lama 3 or open Ai and get back the results do you start doing

15:27things like that that maybe maybe not yet uh we're not using gen for the identity resolution but we are using gen after the identity resolution so basically from you know AI needs the best context and data you can give it and so for us the point here is that if you can give a more robust understanding of who a person of who who

15:50that is then your AI will be able to do far more accurate things so one of the things that we've been working on that we're going to have in the solution accelerators on the Databricks uh site pretty soon is um basically you'll be able to use EMP parity but you'll be able to uh input your product data and description data and then the segments that you build an amperity you can M you can use gen to um take that and uh it

16:13will and generate like custom um descriptions for your website and or or for your marketing materials that are tailored to that group that use the right language for the right people and so I think that like for us it's you know like like one of our big taglines is better data better results and I think that's kind of the way we try to simplify it because it's a very complic complex topic and often one of the things that we do in our partnership

16:35with with Databricks is that you know datab bricks is a very uh technical tool overall like it's very you know kind of code focused interface and there's lots of parts of the of the customers that we serve that are all about marketing and and use cases and business strategy and they don't necessarily know how to write code and so if we can make that data available to them uh then then it will be uh you know it helps them understand

16:57their business better and and so I think that's been uh you know kind of a key part of the partnership as well both the uh ability to because because the things that uh let customers solve the most interesting problems in Databricks like building gen models they have to get their data right first and and all of that's very tedious like you don't want to be standardizing profile data like that's been done so many times and so

17:21our our theory is that if we can help accelerate that and give them the best data asset possible and they can focus their time on things like building Genai rather then Focus you know updating a workflow for you know an email matching algorithm or whatever that is right it it just creates more space for for companies to work on AI and and and other you know exciting Revenue driving use cases so basically you need to get

17:44you're gonna need to send your data or inest data on Databricks I don't know bringing emails from platforms using Fon or any other tool ingested in data breaks to run your transformation and then connect data breaks to emper to do the magic that's that's that's exactly right so uh you know we've worked with five Tran quite a lot and basically a company like five Tran will

18:07will populate the data warehouse with all of the raw data from all the channels and make that easy and then we will connect with Delta sharing and then uh you know uh create jobs with our machine learning algorithms to generate that ID graph and then and then once we have that idid graph uh we have a uh we simplify it by using uh semantic tagging so instead of H having a specific schema

18:30where you have to write code to adapt to use amperity we uh we one of the areas where we do use gen is uh in figuring out tags and so looking at a field and saying what do kind of tags should go on that field is an area that we've augmented uh recently with that because it it's really easy for us to be like that's an email and that one is a revenue field versus a return field versus a you know like like all these

18:53other things and but but once we have those tags then we can generate standardized data sets and so uh you know we have a library of you know 20 or 30 different data sets that we will generate for common use cases so obviously your unified profiles but standardized transactions email op Behavior or you know you know marketing response summary predict predictions about varying different things and so

19:17basically you can just point amperity at your data tag it hit run and the initial model will give you 20 or 30 tables that are kind of use case specific you can you can just say I want householding on this and because we generate that ID graph and we have a householding model you just get a householding model and then you can tailor it if you want to tune it for your company but you get like at least basic you know a version of of a bunch of models and then you

19:40can kind of adapt it to your based on your your goals and that's more valuable than like you know having to write that stuff from scratch especially when every company is solving exactly the same problem like like I just have a question can you get the unity catalog metadata because in unity you can have like comments you can have tags and to populate this automatically on the on parity to help enr the semantic

20:04model you have uh that that that's something that we've got been sort of using as an expanded model so one of the things that we uh uh that we've been trying to take advantage of and focus on here is we like I think a lot of SAS tools like us you know were built on like a Hadoop or something to that effect um and we didn't do that we actually have been using a Lakehouse style open open source format storage

20:27model the whole time and so um the cool part about that is that you know the the project I've been focused on is creating that Delta sharing based integration and then you know one of the first goals of it is extracting as much data as possible out of the unity catalog bit and uh and so I think that's uh the intent is to have kind of a single governance layer where uh they can manage not only access but the

20:50metadata and the descriptions and so we want that to be a a loop because we will populate Unity catalog for the new tables we generate alongside it and so we can uh uh take in the tags from an inbound the inbound side but also um uh um when we're generating new data sets we're uh we're also populating Unity catalog so you can use that Unity catalog bit as your your single layer of

21:13governance and that's been pretty powerful for companies so what's the what's the main architecture you see with your customers do you um do you connect directly to the customer system do you have some kind of reverse ETL to load the data to amperity do you do you primarily connect to like a data warehouse like Databricks to get data what do you see well up until about a

21:36year ago we would always provide our own storage for everything and so one of the biggest pushes with we recently launched the Lakehouse CDP as a new position that's been kind of what I've been at the head of and the big difference there was basically taking the storage layer for imperity and making that configurable so that you can put it in different places and so that allows you know our customers to use data BRS as

21:59their storage layer and that uh that Mak that basically simplifies things from a security stance a lot it also uh um um means you're not you know having to maintain copies in different places and that kind of thing um but as a result you know we actually have an entire framework of sort of SAS etls for getting data from external systems you know and putting it into the storage

22:22layer and so you know you can uh you know we have connectors for things like your Salesforce or you know Clay or uh for a lot of your different marketing channels and uh so we'll pull that raw data in uh for our customers sometimes I mean really our thing is the platform like I think we take the term customer data platform very very literally we are a data platform that has a lot of features specific to

22:45customer data so we have U uh tools for pulling raw data in VI connectors we have um tools for unifying and modeling that data and then we have reverse ETL uh interface for marketers to be able to push data out into their various channels uh so so we kind of you know and then on top of that we we have workflows and change management features

23:08that span the whole Spectrum so the point is like you don't have to use every single thing that imperity does at once but it is uh very much designed to kind of take the most common problem every company has and um um make it you know so that you don't have to you have many tools or do too much solutioning it can work out of the box there that said we have tons of customers who use us and

23:30five Tran and or use us and you know other tools there too so it's not like I think the key is to give companies the option some companies really want to build their own stuff some companies really need that the sort of paint by numbers approach where we can tell them like you know here's the problems you need to solve and here's the tools you can use and if we can offer solutions for that it makes it easier for them so that they don't need a massive team to support and maintain

23:53things maybe one last question I heard a lot that uh things around reverse ETL and and CDP platform and that's you know people are moving toward rever CTL because they want to stop the I don't know if it's the old way of doing CDP I don't I don't even know if that makes sense uh but what's your thought on that oh I mean it's complicated a bit but so the weird part to me is you know I've is

24:16realizing that I've been in imperity long enough for there to be an old way for CDP right like you know prior to imperity there was B basically the model was you would you know ETL used to be a very literal term you was extract transform load and you would use something like Informatica or you would talend or something to that effect and you would write a job that did transformations of move data and it was pretty manual and heavy but and when you

24:40have hundreds of tables you know if it takes you five hours to write an ETL and you have five 100 tables that's 500 hours right like like that's you know that's a ton of work you spread that over a team over time it's just you know it just completely chokes uh even an Enterprise company and so you know our package of like features model is you know was kind of there to solve that

25:03problem but um but since then the I think that there has been a move away from large Enterprise SAS generally like say even Salesforce missed their their sales goal last quarter and they don't miss sales goals like that's kind of their thing you know and so like I think that the current economy and the sort of recent changes in Tech have created a

25:25lot of cost sensitivity and so people really need to understand the value if they're going to spend you know six figures or more on a on a a big um you know uh big piece of software now and so I think that has created a space for more of these Component Solutions and so I think you know like if I'm thinking about it like you want to know what your job is you're used may maybe you've been

25:48around at the company since you were writing informatic etls and like I think that the uh composable style solution offers a similar enough feeling that people have been uh you know kind of open to it but but from my perspective the main composable CDP story is you use fet Tran or maybe air bite to get data into your data warehouse and you use

26:12like sensus high touch you know all the any you know Rudder stack any other reverse ETL tool to get data out of your warehouse but it kind of leaves all of that stuff in the middle that we've been talking about up to your team like you know standardizing it and turning it into something like reverse ETL tools are basically connectors to get data out of your data warehouse and into your marketing tools but they don't do anything to make the data right and so

26:35they need the data to be right before they can be valuable and the hardest problem of all of it is standardizing all that data and unifying that data like it's very valuable like I don't want to write connectors all day that's tedious it's valuable to have connectors but like it doesn't make it a platform and I think it does like if if it takes you know 100% of the effort to build your CDP then like five Tran will take

26:58away 15% and maybe like a you know census will take away 15% but that still leaves you know 70% like and the 70% is the hardest part like and it requires the most you know uh uh kind of longer term technical uh uh depth to really to really build it and you know at an Enterprise company that means you know

27:20time means the potential for change means you know reorgs and re priorities and all kinds of things and there's so many half-finished projects and so you know the fact is I can take an imperity tenant from zero if all your data is in Databricks I can take that from nothing to an initial model in like you know a matter of hours and like no matter how many tables you got and that's that's the key that we're trying to focus on and then you know if they want to use uh

27:44uh different different things for their connectors that's perfectly fine U but I think that that that our big differentiation is that middle bit Yeah and honestly I think you nailed it uh because even if you move toward like a composable uh data platform and you have you know more and more systems then the you really want to have really really good data to start with and clearly identify customers I feel like because

28:06if you have multiple system it's going to like the I feel like the effect of having bad data is going to amplify as much you know as bigger the system grows so well and I think it reinforces silos too you got to think about it that way like if if marketers are using your reverse ETL interface but like that all the data is being prepared somewhere else there's still a silo there there it means that there's one team over here

28:28and so imperity is like you in one UI it's more tabs like the first four tabs in imperity are very much for the technical people and then the last three or four for the marketing people but like the storage layers the same the workflows are the same it's all linked the same and so like it's just much easier to do the other thing that that I think gets missed in most build scenarios is operations people really undervalue monitoring things and like

28:51and supporting and making changes so like if you're using four different tools to build a CDP then the problem is you have to monitor the jobs that are in FIV Tran and monitor the jobs that are in census and monitor uh your your own custom jobs and like either you're going to get yet another tool that monitors all of those across all of them or you're going to have people that have to know which of the four or five tools

29:15you're using to go look at when something breaks and I just think it like and then like if you want to make changes you have to make a change in the source and then also in the middle and then also in the end and that's three different places I just it's like you know uh our model of that is you create a Sandbox and because we're handling that end to end process the sandbox for us is basically a one you know oneclick staging environment so you can make a

29:38draft of your entire tenant you make a change and then you push it Forward I think it's funny because I think the aric has the same kind of um you know ID to have one single platform and to really unify everything and increase the data quality and only now I can see how I translate to you know customer data platform I think it's exactly the same story just one level more toward about you know maybe the business level uh but I think it's really the same ID so

30:01well and I think that's absolutely right and that's part of why we've really been leaning in with Databricks because I think the key here is that I don't want it teams to have to use our interface if they want to use data brecks and I want the data to be the same so they can write a notebook and they and it'll update the data or they can use our Stitch it'll update the data or both in a nice way but like Unity to catalog bit means that they can uh

30:23administer that data from a single place and handle ownership and control in a single place and I think that really powerful and so you get the best tools for the technical teams but also you serve the marketing teams and it all uses the same Lakehouse storage layer and that's the powerful thing like I actually think connectors long term not immediately it won't be a drastic change but like I think connectors are are going the way of the dinosaurs man like

30:45they're done because why would I build a connector into braze if braze could just build a integration into Data breaks and then they just you just they just access the data in data breaks and they you know they you and connect via Delta sharing then like you don't have to have a connector there's no job it just reads the latest data when it goes to send the email and so I honestly think that like it allows like the business logic and

31:08processing to be modular and you can start to build apps but everything has the same storage layer and each one can have its own different compute warehouse for the you know this one needs to be really fast this one needs to handle machine learning this one's needs to be real time you know and you can use Delta tables uh you know for for all of it under the hood with different Computing and I think what you described will speak to many people like having to uh I don't know Implement maintain different

31:32Stacks like one for OBS one for observability one for the connectors Wonder for the ETL and like having those seos it's super super like and it's also like increasing the cost because you need to have an expert for every single tool for every single tool a support of course connect all those tools together like rather than focusing on what brings

31:54the value like out of your business you focus on maintaining different Stacks so that's that that sounds I mean I I I clear I am clearly on your side oh no I get it I mean I can get I can get real worked up about this sort of thing but I agree yeah I mean I think that the hard part is like the whole point of SAS is that in general is that you know if

32:17you're running an airline then you you should be best in class at running an airline keeping the planes you know full and you know dealing with all of your all of that stuff and like you can't also be aof Ware company and I think the whole point is like the more we make it this modular approach the more we're expecting them to be a software company like they were expecting them to know what kind of layer they need to use where and I think that if you know their

32:40tools like imperity or or or five Tran or whoever can adapt to the kind of architectures they want then we can stop them from having to worry about that sort of thing so much and that frees them up to do the other things that are to your Point Drive actual revenue and Caleb can you show us please the platform I'm really curious to see how it looks like you know I have this this data in my bronze dat you this bronze database and so that is you know

33:03e-commerce data customers transactions Etc and each of these has kind of its own schema of data that's available and then you have you know loyalty data point of sale data that kind of thing and so each of these you know has their own IDs I I could buy online I could buy in store I could have a loyalty uh Pro profile and you want to have a clear understanding of who that is and so what what you're able to do is you

33:25connect it to imperity um by going back out to your catalog uh going Delta sharing and then I've created a share already for the demo here but uh you know in that demo is the e-commerce data loyalty and point of sale data so I'm going to go to recipients and I'll add a new

33:47recipient and then that gives me a link to download my share credential so I downloaded my credential file there and now if I come amperity um so amperity is basic UI here let me zoom in a bit um is you have sources is accessing data so this is where we're going to set up our connection to Databricks Stitch is unifying it on down and

34:11so we'll walk sort of top to bottom but very much like uh the like we've been discussing the um the point here is that we want to be able to pull the raw data in uh run feed that into our Stitch model which generates standardized cleaned up data feed that into our enrichment models which generate uh you know a unified clear data and then provide that interface at the end and so

34:34like ultimately if you pass in these types of tables you'll get you know uh you you'll have your customers transactions Etc you tag them run stitch stitch will output a library of tables based on the available tags and then you can tailor those models and turn them into sort of cleaned up uh useful data and so uh um if I start here at sources and I go look at that uh down here what

34:57we've called the feature for our uh data sharing connectivity is Bridge so I'll give that a name and then I'll need to upload that credential um here we go this one yeah it's the last one yeah okay oh yeah you're right because I

35:19already had one I had to get the okay so there we go and then now it's like that'll connect to the Via Delta sharing and you can see those same tables I was pointing out in data basically just letting me know hey you know these are the ones I want to be able to use I can check that um I've already set this up so it's going to give me name conflicts but basically it's telling me hey these are the tables we're going to register for imperity to

35:42be able to use um uh I've already set this up kind of cooking style here so I already I already have a a version of this working but basically once you have that then we apply a set of tags and so if I come in here um and look at you know you see all the same tables e you know e-commerce transactions itemized point of sale Etc so I I'll open that up

36:05and you know I have the option to tailor it a bit if I want to filter it down or something but this is where you know all of these were just guested but like the the point is uh we have a library of these semantic tags so you know profiles uh you know first name last name your full name your your generational stuff you know addresses phone numbers Etc but

36:27we also have tags for a ton of different data models like products and uh transactions at the item level and you know uh uh compliance requests and loyalty profiles and email events and so this is you know not a very visual bit but again the kind of Point here is these tags uh when applied to these types of tables output a bunch of different models based on those tags and

36:49so uh what we do is we will tag all these uh different fields and uh then we'll activate that um oops and then that gives us this kind of a view here now we'll do the same thing for uh like itemized transactions here um and then you can see you know I've tagged that with transactions at the item level you know discounts whatever

37:11that the different fields that are represented here so once we have those tags oh go ahead just one question kale for the auto taging have you built like your own is it a netive behind scene who go and understand the metadata to tag the tables or is it a model like how does it work so there's a couple of different ways it works um so if we are connecting via Databricks we can pull

37:33them from the unity catalog um but the but but often what it what will happen is when we create the data we actually have connectors to many different things we call them couriers and so you know you can see that there's a whole bunch of these you know Adobe and whatever else in here um and so some of these we have information from the schema that we know from the their apis but most of the

37:55time you know a lot like one our number like if you look at the top examples here Databricks SFTP you know the they who should not be named and then S3 and blob storage but like you got to like most of these are not uh specific apps they're like just storage on the top here there are most common ones and so what that what what what this will do uh there is it will look for um oh I

38:18haven't done it here but it'll look at the first 10,000 rows and it'll feed that into a process to sort of predict them and so when you look at it and you see you know um all these different tags on it like basically these will look at at the name the first 10,000 rows and and and in a given uh uh model and it'll sort of predict what those are and so

38:41the common thing to do here is like you can see this one is you know uh uh you know has this ID field here like you might want to actually change that uh format but like you can update the format you can update the semantics and so it like basically if it gets something wrong and like you have a field the most common thing is that you'll have a field that has no real discernable name or is sparsely

39:03populated and so if you have something that's like Evar 15 or something then we won't be able to necessarily guess that unless we look at all the data and see that it has an email address and so you know it's it we either pull it I guess to summarize we either pull it from the metadata in the connector um or we look at the first 10,000 rows and use our own uh uh kind

39:25of processes to guess and then you can tweak it before you hit lock go does that make sense yeah thank you I can't see the screen and while I'm sharing so okay uh just for the sake of time here so once we've tagged it we kick it off and we run Stitch and so this is our Stitch report and what this is telling me is you know 7 million records uh my my my you know this is a

39:48cleaned up instance for telling the datab Brick story but like this is our normal sales instance as 130 million you know and so you know 14 minutes for that Stitch you know the but basically the whole point is that you know across these three systems you had you know 2 million IDs 2.6 million here 1.8 million here and overall that was 7.1 million unique IDs across those systems and and

40:12when we stitch we found 4.8 so that basically means within the 7 million rows we found 4.8 million clusters of data you know which roughly translates to a person although sometimes if you don't have enough information you can get a lot of Singletons but you know that's something you'll tune and then you can look at each of these and it'll tell you overlap with other data set sets you know what semantics were available since this is fake it's all

40:34overly populated but um you know the but what really the kind of key element here is it generates this graph and so uh you know in this example here you can see you know Alexandra uh you know she's got four different duplicates of my e-commerce data uh loyalty uh profile and a and a point of sale profile couple of

40:56different email addresses this one slightly misspelled sometimes she's Alex sometimes she's Alexandra maybe at some point she got married her last name changed you know and so the point is we compare all of those things and then we get a graph and that graph is you know we have hundreds of these things and you know or you know there thousands of them we've generated uh they all kind of look similar when they're fake here but um the kind of key element is

41:20that you can actually interrogate the graph so a lot of people do match and merge we do match separately and we generate the graph and you can use that graph for any number of use cases but like in this example you can see we compared and contrasted this person we decided that give him a a score of 4.4 4.14 out of five and you can actually like dig into it and this transparency of being able to see what rules were

41:43applied and what weights were given is why you can use machine learning for identity resolution like without the transparency it's kind of pointless like you can't it's not confident enough to use um and so now you have a graph with a single ID and then each of these nodes represents one of their profiles in a different system so if I go over to my like bigger instance here you can see you'll be able to see much larger graphs

42:07um you know and you get kind of cool art out of it that's one of my favorite bits we have an entire channel for like cluster art it kind of generates all these like you know very odd shapes fun so what's the what's the use case behind that would you would you have human review maybe your top customers just to make sure you don't mess up with these one so typically what we do is we apply a default model and then uh we use a

42:31library of of Stitch qu QA queries to tailor it a bit for the customer and then uh then they can feed in labels if they want or like block lists is a very common example so like you know within the settings there's a ton of different tables that I I won't get into too much but you can set the confidence thresholds and you can order the different blocking strategies and

42:53there's a bunch of other like little things on here and there's even some more advanced stuff back here and so basically you know we work with the customer we apply the default model and then we tailor it based on their needs uh and that what they know about the data and kind of what their use cases are so a common example is like if a loyalty ID uh matches then you can give that a tag that says you know don't ever

43:18uh second guess this and so you know you know and it allows you to sort of have a combination of rules based and machine learning at the same time cool uh just in the interest of time you know we have our ID graph the next step is we turn that into a bunch of uh unified models and so we'll generate you know like the a single profile a

43:41customer 360 table best available pii rolled up behaviors from all the different systems um and we also generate a library of these standardized tables and so like again my goal was to you know generate all of these useful tables um now you can see these down here where we have like you know standardized transactions from across the different systems where you have an imperity ID and all of these simple

44:03fields that you can use to understand your transactions regardless of where they came from you have your transaction attributes so these are simplified flags that are often useful for like a reverse ETL interface so simple flag for one undone purchasers versus multi-brand purchasers you know uh uh rfm scores thing you know Behavior summarized in different time frames uh and these are

44:25all just out of the box Concepts um we also have predictive Concepts so you can have like predicted probability of purchase uh you know predicted you know we have predictions for where they'll churn we have predictions for product Affinity um where you know like you can you can guess based on the type of customer they are that what type of uh of product they might want but we also

44:47do things like create standardized views for you know contact information op status whatever and all of this data becomes available in Databricks and so you know the kind of key element here is getting it back into Databricks um I've been talking a bit about do you guys have any other questions before I I move forward on it all clear okay yeah maybe one quick

45:10question on the on the business model are you guys walking at the RO level like five TR or is it something else um sorry the uh the business model meaning like what models we output or like no no I mean like how do you like what's the billing oh the primary role of our pricing at the moment is based on the Stitch output so it's basically

45:32this number right here is our biggest modifier because uh all of our jobs uh you know generate some sort of compute uh but like uh this is the one that's most expensive and so that's the primary component is how many profiles we're managing which roughly translates to how many imperity IDs um you know we're we're we're evaluating alternative approaches uh because we do want to make

45:54it a little like more you know a little more accessible for smaller companies to use but um the biggest thing for us is how many profiles and then how many transactions and behaviors overall so it's it is sort of row count based uh but it is uh kind of biases towards the uh value that they are getting so

46:16the actual profiles that are being managed charge for like how many sources you have we don't charge for how many uh uh outbound destinations you have so like my other tenant has a lot more destinations way too many that's why I'm not demoing from it but you can you know have data being sent to your your bra setup or you know and your Facebook and your Google cloud and your whatever and

46:39the kind of point for us is that you know that if you're using that that means you're getting value from the data which means you're likely to stay with them parody and like sending data outward isn't very expensive so we don't really charge for how much uh how many of these you use like this in my demo tenant I have hundreds of different you know things for very specific demos demo stories but like um the so like the key

47:01model is just data we're managing for you awesome okay so uh just for brevity uh you create an outbound share to get the data back uh that this is all the tables that we just generated for you um and so you know you select which tables you want and you download a share uh you know a a credential uh and you basically

47:24you know import that into Delta sharing the same way we did the first way and the end result gives you um you know back in your catalog you'll see you know oh look there's a there's Delta share down here and now all of a sudden I have all these tables that are worth using you know that I can go build AI on or custom models or whatever uh there's a whole marketer interface and everything too I'm but you know I don't want to do

47:47a 40-minute [Laughter] demo no that was amazing I love I I love it honestly uh I feel like it's super easy uh um like I I'm going to say it again but if you if you want to try to do that yourself I would really reconsider and try to see if you can buy something inside uh and I love like the you know the how easy how easy it is to get the

48:11data in and out like just using Delta share I think it's really powerful and yeah like if we can make the connector slowly die to make everybody's life easier would be would be amazing I mean I I think it's exciting we're going to be able to have a modular uh set of applications that run on Databricks using this kind of model and I think that it'll be it'll be great it's easy to extend yeah yeah my use case down stream on the on the data would be

48:34amazing and one more thing before we end uh the call or the recording do you have any trial period that we can attach in the recording like make sure like if someone is interest how they can get started or to whom they need to reach so uh so data works impair.com is a email list for that at the moment we are beginning to take uh request for that I

48:58think the so like like a little background here is that imperity was very much built for the Enterprise originally and and and we didn't even have a UI for a couple years so like it's you know I mean it's been a long time we've got we it's been a long road there but we have been U um uh you know kind of Shifting towards a more product Le motion here and so we're we're experimenting with a consumption model

49:21which would allow people to do pay as you go where like you know because not everybody wants to most of our customers up update their data graph uh several times a day or once a day uh they want it up to date constantly but some people just want to do like one match and they don't you know and so they so we want to be able to open the door for that kind of thing and so we're we're experimenting with a consumption model for that and then we're cons we're also

49:44uh beginning to roll out a a sort of I I won't say beta I'll say Alpha uh free trial concept and so uh but if you but if anybody's watching this and is interested and wants to give AER a try uh you know datab bricket and.com is the through email for that and so please feel free to hit us up on that and then of course you know uh I'm I'm looking forward to kind of that transition into being able to see what

50:06people build with it because I do think that that ID graph as an asset is just you could use it for clean rooms you could use it for compliance requests you can use it for AI you know you can use it for any number of things uh there's something I didn't have a chance to show is our sandboxes but also our like I we have you can output that uh that ID graph in uh you know in a format pre-built for graph databases which is

50:30like you know super useful for a lot of modern cases so I just think that there's going to be a lot of great uh um um you know I'm excited to see you know it get out there without the services motion Associated and and I'm really you know hoping to see uh what people build with the identity graph over time that sounds good awesome no free trial at the moment but

50:52you know email us and I'll I'll take care of you that's works well as well yeah yeah thank you so much like that was uh anation demo uh all the features you do have on the platform how it has been built how the platform is evolving and you give us a sneak peek of what to

51:14expect in maybe next couple months thank you so much any other words Quinton Caleb no thank you Caleb it was amazing to have you yeah I appreciate it guys this is fun times and uh you know I've been watching more and more of your videos I'm excited to kind of see this grow and you know nice work with building this out thank you thank you thank you byebye hey take care