Building AI Agents with Unity Catalog Tools in Databricks

0:00[Music] all right thank you Patrick so all right where are we all right so I work for a cookie conglomerate that has a bunch of franchises and I want to create an AI agent to help all my franchise owners improve their business by allowing them

0:23to analyze customer data allow them to create marketing campaigns and and analyze and develop sales strategies and so one of the things that they'll be able to build with this AI agent is an Instagram ad campaign where they can promote the bestselling cookie in their franchise and the AI agent is going to create an image for the Instagram ad as

0:45well as a caption that's going to capture the hearts and minds of all of our cookie lovers to really Drive sales so I gave general intelligence to my franchise owners with the off-the-shelf model and it was giving good results but they were too generic and they weren't tailored to our business or the individual franchises and this is where the Mosaic AI platform

1:08comes in Mosaic AI is going to allow us to extend this general intelligence with our Enterprise data so that we can have data intelligence in this demo we're going to build uh sorry in this demo we're going to build uh an agent that's going to use the unity catalog tool that Patrick just

1:31mentioned so in this architecture we're going to leverage these UC functions that can now be leveraged as tools and UC functions can be SQL functions that access your data warehouse they can be python functions they can be model endpoints and they can even be remote functions which are going to allow you to call external services like slack

1:53or uh email or even file a ticket if you need to so to build all of this we're going to use the Mosaic AI platform and with that we're going to go ahead and jump right in so here I am inside uh so there's three capabilities that we're going to use inside of Mosaic AI to actually build this data intelligence so the first is we're going to use our Tools

2:16catalog to actually build the data intelligence then next we're going to build uh and understand our quality with agent evaluation and then we're going to be able to debug and improve our quality with the mlflow tracing capability so with that let's dive into Data breaks so here we are inside Unity catalog you can see that I have some functions that

2:38I'm going to use as tools and these are governed alongside my AI my unstructured data and my structured data so to help demystify what a tool is we're going to go ahead and click into our franchise sales so you can see in here it's just a simple SQL query that's accessing my sensitive transaction data and this is where it's really important that you're tools are governed alongside your data

3:02because only the people who have access to this underlying transactions table are able to successfully call this tool and that's why we need this centralized governance across data Ai and tools so some of the other tools that we've created that are leveraging our Enterprise data are in here as well so franchise by City And by country are just like helper functions to help me get the sales data and then this

3:24franchise reviews tool is actually grabbing customer reviews from our social media sit so all of these tools are leveraging my Enterprise data so we're going to go ahead now and extend a base model with these tools so I'm going to come over here into the AI playground and we're going to jump in and then from the AI playground I'm going to select a

3:47tools enabled base model and so I'm going to you can tell it's tools enabled because it has this little icon on the right so I'm going to go ahead and select llama 3 from here I'm now going to add hosted tools so these are my unique catalog tools that are hosted inside the secure and scalable data rcks environment so in here we're going to access the tools that we just showed you in the AI schema so I can use the

4:10syntactic sugar to grab all of those tools and then my marketing team has created a tool for me so I'm just going to copy paste it real quick because it's kind of long so we're going to put this in here and this tool is going to generate an Instagram image using the Shutterstock image AI model that Patrick just announced as well as a caption so now it's time to actually test this

4:34before I forget we're going to crank our temperature down to zero because this is a live demo uh and then now we're going to quickly copy paste a prompt in here so this prompt is going to say hey send marketing an Instagram post with an image and a tagline for the best-selling cookie in the San Francisco store so we can increase our sales and show that we listened to customer feedback so we're

4:56going to go ahead in here uh and click uh oh we're doing it live uh well that should have unfortunately worked hold on we'll try again so we'll go back in here we're going to add our tools AI star and then we're going to

5:20come in here and add our marketing oops oh it's now in our auto complete in here uh and now let's try this again H unfortunately it looks like there's a connection with our H unfortunate uh what is this catalog ret oh oh oh I'm so silly I typed in the

5:44wrong name of the function you all should have caught that for me I need to type in retail. thank goodness for error messages so we're going to come in here and now we're going to add this in uh and now we're going to go ahead and do this this thank you thank you all right so what is happening here

6:07is going to be a little bit magical so we're gonna come inside oh hold on H oh my goodness all right we're gonna have to bear with me we're gonna have to do it all again all right we're coming in here uh coming in here sorry uh this is why you don't do it live all right retail p. a all of the functions in there we're going to add in our marketing tool that's going to generate

6:29our Instagram ad all right we're in here we're going to send the prompt and we're going to make sure it's temperature zero which is what we kind of forgot there temperature zero live demo all right here we go now we're going to send this and what's happening is going to be kind of magical so as we come in here you're going to see that llama 3 is going to do Chain of Thought reasoning so it's going to

6:51figure out which tools it needs to call in order to execute this so you can come in here and say oh the first thing I had to do was grab the franchise ID for my San Francisco store the second thing is it needed that to be able to access our franchise sales because it's trying to identify what that best-selling cookie is then it's going to grab all of that sales data in here so it grabbed all my

7:14sales data and from here I identify the bestselling cookie is this almond biscotti cookie from there it said hey we need to show that we listen to customer feedback so we need to look into our customer reviews tool from there it's going to ask hey what do customers like about this biscotti cookie it's going to say oh they like the crunchy texture and unique flavor and it's going to send all of this to my slack tool that's going to generate this

7:36Instagram image and caption and send it to my marketing team on slack so they can review it before we post it on our social media and so now the moment we've all been waiting for is let's see what it actually returned so I'm going to jump over into my slack and this is the image that was generated by the Shutterstock image AI model that shows our image

7:58biscotti and and then you can also see that it creates this customized uh caption where it says our customers Rave about our biscotti for its crunchy texture unique flavor and perfect coffee dipping quality and so this is what has generated and this is how you can use data intelligence to extend your general intelligence to improve a base model now

8:21what happens if I remove the intelligence so I can come in here kind of like we did earlier and we're going to remove all of these uh Enterprise data enabled tools and we're going to run it all again so now we've taken away all of our Enterprise data access from this and now it's still going to generate an image and it's still going to generate a caption according to that prompt but it's going to be much more

8:44generic so as we jump back into slack and it's going to show me my new image soon com down here okay successfully sent here we go so this is the image that it now created so all it had to go off of was that it's it's a cookie that's in San Francisco and so it tried to create some kind of cool Instagram ad yes

9:11and and if you take a deeper look at the caption it's very generic so it just says our bestselling cookie is backed by popular demand share your favorite cookie moments with us and so not really tailored to our specific business or the franchise or using our Enterprise data at all and this is why data intelligence is so important so we just showed how you can use the Tools catalog to extend

9:34your general intelligence with your Enterprise data to create data intelligence but how do I know that this agent is high quality so the way that I know it's high quality is I'm going to use agent evaluation and mlflow tracing tools so agents are really hard like how do you know if what we just did was good or bad so we're act and there's so many different things you can do with an agent as well so we're actually going to

9:57have to launch a pilot program with a sub set of the franchises and give them the agent evaluation review app so this review app is going to allow all of your franchise owners to interact with your agent whether or not they have a datab breaks account and then it's going to allow them to give feedback on the response so they can come down here and they can say yes and they can explain why or why not uh that this answer was

10:21good and then they can go ahead and click done and submit this feedback this feedback that they submitted is then logged in a Delta table in unity catalog in your account that you can then build an evaluation data set off of get confidence that you can go into production or as I have done I enabled Lakehouse monitoring on top so that I can observe how my pilot program is running so you can see over here the

10:44different franchises that I set up this on and here is me tracking their negative scores with the agent over time and you can see that something is going terribly wrong with the Los Angeles franchise they're getting a lot of negative feedback on the agent so if I scroll a bit further to actually investigate their ratings and what questions they're getting they're rating poorly we can see that their actual

11:08feedback that they're giving me on here is that it's returning IR relevant reviews so it's returning reviews from San Francisco stores or non-la stores to them or it's even hallucinated that there's a liberty chip cookie which we don't sell one of those at the cookie conglomerate so we're going to need to dive in deeper to figure out what's going wrong with our quality here so we're going to use mlflow tracing to get deeper and figure out what exactly is

11:31going wrong so I'm going to jump inside of a notebook where I've actually queried my assessment logs in here and in here we have captured an mlflow Trace automatically for you so mlflow is a popular tracking API for geni and machine learning experimentation and deployment and so we've extended it to now work with compound AI systems where you can now trace your input to the

11:54system and how it's transformed as it goes through every step of the system along the way to actually create that output in the end so I can click on one of these traces and it's going to open up this stack view if I click the top of the stack you can see the question that was sent to the system and the output so you can see this is the one saying what our customer saying about Liberty chip and then this is the absolute

12:16hallucination which we learned from the review app where it's saying hey customers are raving about this cookie and it's out of this world delicious but we know there's no such thing as this cookie so we need to figure out what went wrong so we're going to go over into our stack and we're going to dig deep deeper into it and so we're going to go to our first tool that was called which is this customer reviews tool as we go in here we're going to see what

12:40the input and output of that were and if I go here you can see okay it's saying that the Liberty chip cookies out of this world delicious so this is definitely where the problem's coming from so we're going to need to dive even deeper into the stack and when we get down here we're going to get into our retriever so this is the thing that is actually retrieving our customer reviews and so if I look and now because of

13:02tracing I can actually see the exact reviews that are returned and so I can see this review is saying hey the staff is warm and welcoming the store was spotless and the cookies were out of this world delicious so what's happening is my retriever because I can't find anything about this Liberty chip cookie is just returning random reviews and so I'm going to need to do two things to fix this the first I'm going to need to

13:24actually increase my criteria threshold for relevance on my review app retriever so it says like hey don't return reviews if the relevance is below a certain threshold and then I'm going to need to do a little more prompt engineering to make sure that if the context that's given to my model isn't relevant to the question that was answered don't just summarize that context so I've gone

13:47ahead and I've already made those two fixes and I've redeployed my agent and so now we've redeployed the review app as well sent this out to my franchises and in my pilot and so now they can say what are customer is saying abouts the Liberty chip cookie and so if I type this in here you can see that now instead of hallucinating it's saying

14:10that Liberty chip cookie is not mentioned in the reviews it's possible that's it's not sold in these stores which is exactly what we wanted to say when this happens all right so in this demo we just showed how you can use the Tools catalog to build data intelligence by extending a General mod or general intelligent model we saw how you can use

14:32agent evaluation to actually understand your quality by getting your agent into the hands of your franchise owners even if they don't have datab breaks accounts and allowing them to give that humanin the loop feedback with thumbs up thumbs down and then we used mlflow tracing to allow you to debug and iterate on your quality to improve your agent so we talked a lot about many

14:54different things in this talk but there's three main key takeaways that we want you all to really gather about the Mosaic AI platform the first is that we have to move from general intelligence to data intelligence and the way that we do this is we augment general intelligence with your Enterprise data and this is going to give you much better insights into what's happening in your business as well as it's going to

15:17improve the quality of your applications and we saw this with the franchise cookie agent where the images and the captions for our Instagram ad campaign were much better once we gave it access to that Enterprise data this second thing is that you can also improve quality by moving from these monolithic models to modular modularizing them down into AI compound systems where now you

15:39can specialize each step in the system to improve your quality like we saw with the fact set use case and also in many cases will also improve your latency and lastly the Mosaic AI platform is the best platform to build high quality compound AI systems we have thousands of customers using uh the Mosaic AI platform today