Agent Bricks Demo: Building and Optimizing AI Agents

0:00Thanks, Hanlin. All right, so for those of you who tuned in last year, we're moving on from cookies to energy drinks. So, in this demo, we're going to build two agents with agent bricks. The first one, we're going to build a knowledge assistant for our energy drink company's R&D department to help them ask questions over their different formulas and recipes, and we're going to use automated evaluation and agent learning to help them improve the quality. Then

0:24we're going to take that highquality R&D agent, combine it with some of our other departments expert agents into a multi- aent supervisor system so we can answer our CEO's toughest questions. So with that, let's go ahead and dive in and build this knowledge assistant inside of Databricks. We're going to come over to the left nav to find agent bricks.

0:46Hold on. There we go. And we just have to click on this agents tab over here to get to it. Here you'll see there those four tasks that Hanland mentioned. So today we have these predefined agent workflows for four of the most common agents. More will be coming for the R&D department. We're going to build a knowledge assistant because they have a bunch of PDFs that they want to ask questions over. I went ahead and already

1:08configured one of these, but you're going to see how easy it is with our declarative approach to build one of these agents. So all you have to do is in natural language describe the task that you want your agent to accomplish and then you just need to configure your knowledge sources. It could be an existing vector search index or if you just have a bunch of files in a UC volume that works as well and that's what I have here is a PDF of a bunch of

1:30recipes and formulas in my UC volume and agent bricks is going to automatically index it into a vector database for me. So with that you just click create agent and then you have an agent. Uh, and the way that you can see that is that you will automatically have an API endpoint built for you that then you can integrate into an application or you can immediately test it. So, let's go test it out. So, a common question my R&D

1:53department asks is to ask about the surge sugar cap formula or to see if their formulas exceed the surge sugar cap. All that really means is they want to see if there's so much sugar in the energy drink that it overwhelms the flavor. So, we're going to go ahead and test it out. So, we're going to see does Does spicy mango exceed the surge sugar cap?

2:18And we're going to go ahead and ask that question. Typo. Well, now you know it's live. All right. So, in here, you're going to see that already you have your knowledge assistant built and you can already start querying it and you had that. That's the benefit of this sort of declarative approach that we have. The other thing that you'll notice is that it provides citations for you. So, this helps you build trust in that knowledge assistant that you've built. It's going

2:40to site its sources as well as link to the various documents that it used. So just like that, we now have our knowledge assistant built. But let's take a look at this answer. And it says, uh, it's basically saying if you have that information, you can make the comparison. But it sounds like it doesn't really know what this surge sugar cap is. And as the developer, I don't know if it doesn't know this

3:02because there's a problem with retrieval or maybe I just don't have any documents that have this information in it at all. So this is where automated evaluation is going to come into play. If I go back to my agent, you can see that every single agent is configured with MLflow tracing automatically set up and MLflow trace is automatically tracking every incoming request, every outgoing response and all

3:25the inter agent calls in between. You can then also see that my agent bricks has created and enriched every single trace with automated evaluations. So you can see here it created custom metrics for my task and is giving me these sort of automated insights right away without me having to go to a subject matter expert for a manual review. So now I can

3:47instantly get a signal on my agent performance. You can see that it's still loading the judges uh still like running the judges here. So I'm just going to go ahead and look and see if other people have asked about the search sugar cap. You can see it's not doing well on any of these questions. I can then click inside, go inside the details. We can see, you know, reranking is run. We can

4:10see which documents are returned. And then sure enough, you can see none of the documents that returned are relevant. So it doesn't know, none of our documents have any information on the surge sugar cap. And this makes sense because I gave it a bunch of recipes, but this is domain specific jargon. All right. So with that, you just quickly saw how agent evaluation is going to give us these custom metrics

4:32out of the box so that we can quickly understand our agent performance and how MLFlow traces are going to allow us to root cause what the problem is. But how do I fix this? Like Hanland mentioned, there's so many different knobs that I can turn and so many different ways that I could go about fixing my agent performance. But I want to remove the guesswork from that. So to remove the guesswork from the equation, I'm going

4:54to use agent learning. Agent learning is going to allow me to use natural language guidance to steer my agent performance in the right direction. So to get to that, we're going to go back to our agent and we're going to click this improve quality tab at the top. The way that agent learning works is I'm going to give it some example questions and then I'm going to give it natural

5:17language feedback on what the response should look like. Now the we found that the more feedback that you get, the better the system does. So, we created this generate questions button, which is going to synthetically generate a bunch of questions for you based off your configured knowledge sources. Unfortunately, we do not have time for that today. So, we're just going to manually add one for that surge sugar

5:39cap question. So, in here, I'm going to go ahead and paste in this question. For those of you who noticed, it's a slightly different question than what we asked in playground. We did that because I want to show you that this is generalizing to the system. it's not overfitting to a particular question. And then I'm going to go ahead and give it my guidelines. So I'm going to use natural language to teach it what the

6:01surge sugar cap is. All you need to know from this is that it's going to look into the flavors archetype and look at the grams per sugar in a 12 oz can. Other than that, you know, we don't need to know. All right. So with that, we've now created natural language guidance. So agent bricks is going to take this natural language requirement that we gave it and it's going to convert it into a technical optimization. So it

6:23might change my retrieval, it might change filtering or it might even change the prompt or combination of the above. So let's see if it actually worked. So we're going to go back to the playground where it wasn't able to answer the question and we're just going to go down and regenerate the response. And this is where the magic's going to happen. So I can even peek inside right now into the agent thinking. And when I zoom in here,

6:45you can see that it already now knows that it needs to identify the archetype. Was too fast for me as well as the grams per 12 ounce. So you can see it already incorporated the learning that we gave it. And then now it is able to answer this question correctly. So that's how you can use agent learning to skip all of these sort of configurations with these low-level knobs and just use natural language to help improve your

7:08agent performance. So now we just have our R&D knowledge assistant. Uh, thank you. Thank you. Pretty cool. So now we just built a knowledge assistant for our R&D team and we were able to understand our agent performance with automated evaluation and we were able to improve its performance with natural language. But what if I want to

7:32ask a more complex question? So, in fact, my CEO called me into her office and she said, "What energy drink flavor should we build next?" And this is a super complex question and it's going to involve a lot of research. I'm going to have to go to my marketing department and figure out which flavors are trending on social media. I'm going to need to go to finance and figure out how much is all this stuff going to cost.

7:54I'm going need to go to R&D to figure out do any of the flavors have allergens in them that's going to impact our market sizing. If I were to do this today, it would involve a lot of cross team collaboration. I would need to figure out which person and which department I needed to talk to. And we might not even be in the same place or time zone. But what if each of these departments had their own expert agent?

8:17And that's where Agent Brics's multi-agent supervisor comes into play. Agent BRIC's multi-agent supervisor is going to allow you to orchestrate together genie spaces, agent endpoints, and MCP servers. This means you can ask complex questions across your structured data, your unstructured data, as well as any tools that you have in order to answer complex questions. So, let's go

8:41take that R&D uh as agent that we just built and combine it with other expert agents across our departments, put it together in a supervisor, and then use that to answer our CEO's question. So, now we're going to go back into Databricks. We're going to go back to our agent bricks landing page. And now we're going to click on the multi- aent supervisor. Just like we saw with

9:03knowledge assistant, there's a declarative approach to building all of these agents. So I just need to give it a quick description in here of what it's going to do and the tasks that I want it to solve. And then I just need to configure my agents. So we're first going to configure our agent endpoint. So we're going to come in here and get the one that we just built for the R&D team. Then we're going to configure a genie space. You'll also note that you

9:25could select an MCP server in here as well. Uh, so now we're going to hook it up to our marketing. I lost it. Oh, I didn't click the right one. Sorry. Genie space. Now we're going to hook it up to marketing. And then we're going to connect it to our Genie space from our finance and bisops team in here.

9:47Finance. Too zoomed in. Here we go. All right. And with that, I just had to click create agent. So super declarative way to do this. I just said what the task is I wanted this multi- aent system to do. I configured a bunch of agents together. And now I have an endpoint that I can query uh in line in here to now test how it works. Now this will take a couple seconds to build which we do not have.

10:10So I went ahead and built this already. Exact same configuration which you can see here over on the left. And now we can ask it our CEO's question. So we're going to ask it which energy drink flavor should we pilot next? How long is it going to take to develop and what is it going to cost? You can see that our supervisor agent sorry chugging along here. So basically what's happening is taking this super complex question that

10:32we asked. It's going to break it down into smaller sub questions and then it's going to figure out which of the sub aents it should delegate each question to and which order it should call them in. So it might take the output of one agent and put it as the input to the next. So let's kind of see what's happening here with that. So for example, you can see here the first thing it said is like, oh, we need to figure out which flavors are trending on

10:55social media. So it went in and asked my agent uh marketing that. And you can see here it returned cucumber apparently. Then we can go in and pass cucumber into the R&D recipe to say, do we have any cucumber based concept flavors? Which then it returned, yep, we do. We have watermelon cucumber. And then it's going to call bisops to see like what's the development timeline and so on and so forth. So, it's going to keep on asking

11:17this for several seconds. And then it's going to generate a report, which to prove that this is accurate, you can just scroll up and see many more calls later. It's going to return a launch report that looks something like this. And this is pretty incredible. In fact, let me zoom in so you can see exactly this kind of content. I would send this to my CEO. It's basically going to look through it. Did a market demand analysis seeing based on social media sentiment,

11:40watermelon and cucumber are the most active. And we do have a combination for this. It's going to look at allergen safety. It's going to look at ingredient considerations. It created the development timeline for me, telling me it's going to take four to six months from idea to market for this. It's going to tell me my cost considerations. It even looked at my changeover costs on my different factory lines. And it's going to give me next steps in here. So, this

12:02is something that now in a couple seconds, I was able to generate a report that I can share to my CEO so that we can make complex datadriven decisions. And so before you know it, you'll be able to launch the coolest new energy drink or whatever you have for your

12:24company out there. And so that was just a quick preview of agent brakes. So you saw that we were able to build quickly a knowledge assistant and get it to high quality by using automated evaluation so we could understand our agent performance as well as agent learning where we use natural language to actually improve the agent performance. Then once we have all of these highquality agents, we're going to need

12:46to stitch them together with our various tools and each other so that we can answer really complex questions across our organization. And that's what the multi- aent supervisor allows us to do. So, we're very excited about Agent Bricks and what people will be able to achieve with this and we can't wait to see what you all build with it. And with that, I'm going to pass it back to Hanlin. Thank you all.