Genie Agents Setup: Metadata, Sampling & Accuracy Tuning

0:04Hey, I'm with Quinton and today we will talk about AIBI Genie. But first, make sure to subscribe to the YouTube channel and the newsletter. Hey folks. All right, so you stop promoting yourself and jump let's jump in the topic. Let's talk about uh the gen room. How is it working and who should be using that? Yeah, that's a good question. At datab bricks they have the goal to make the

0:26data and AI accessible to everyone and it's the main reason behind building AI gen which makes it possible for nontechnical user to ask questions as they would do to uh a technical uh person and it will gr give them relevant uh and accurate response directly from their data and I prepared a genie room

0:48to show you how it looks like and how you can get started quickly. Just keep in mind that you need to have select permission over the table and can use permission over your SQL warehouse. And over here I'm using a sales data for a company called NextGenLakehouse. We have three tables accounts user and opportunity. And here I just ask a

1:10question. Give me the pipeline amount by region. And you see I had the question the response quickly. APAC 153 ama. And we have also a graphs that help us understand this data. Of course, you can ask any questions is text to SQL. Who is my best rep? For example, who is my best uh sales rep? And again, I will

1:32have the response quickly, which will avoid having to annoy someone who knows SQL in order to have this response. You're it's basically uh self BI tool. And you see over here the best rep is called Alejandro Palwin. That's it. But uh Quinton, in order to use Genie or how

1:54can I what are the tips and tricks I need to apply on my gen room in order to improve the accuracy of the response? Great question. So uh having a good accuracy is key because you want to be able to trust your gen room. So you have a few rules like everything I give it to your first gen room start simple you know just add a bunch of tables in here

2:16maybe three four tables that's it try to avoid tables having many many you know field and column uh with crazy column name it's not going to help so start small and then once you are ready and the gen room is created you have a few tips that you want to make sure you have so I'm going to show you that all right so I'm going to reuse the same uh gen room super easy so you see the three tables that we have here so the First

2:39thing you want to check is the table metadata. So I'm going to show you how to do that. You can go in the catalog and you see and here I have my three tables, the same tables I'm using for the gen room. So the first thing is you want to make sure you have all the AI generated. Well, you want to make sure you have comments everywhere in your tables. If you want to go fast, you can just use the AI generating description.

3:01That's going to generate the comment for you. Make sure everything looks good. You can accept and boom, you're done. And you also want to make sure you have the same for all the comments in the field. Same thing, you can use the button to generate all of them and then, you know, double check it's okay and maybe improve that a little bit to increase the accuracy. I I think it's really key to have good comments, especially when the field name is not

3:23really relevant. If you have technical field names, make sure you have good comments explaining what it's about so that you have a good result when Jenny queries the table. So that's why just make sure you have all the comments. And Quinton, I see something. I see in your data that you have primary key and foreign key. How can they help Genie? Yeah, exactly. So that was my second point. Make sure you have all the primary key, primary key and foreign key

3:45define. So you can see them here. You can also click on view relationships to see to see you know all the tables relationship. It's super easy just you know basic SQL query. You create the foreign key in SQL add them and it's going to help doing the joins. Again, it's super useful especially if the column names are a little bit strange and you know the join is not obvious. It's really going to help Jenny to know how to you know join all the all your

4:08data sets. So that's the second point and the third point uh which is going to be released soon is if you go on the Jenny room and you click on the data if you click on the table name so you can see all you know the table name all the fields with the description and you will have a new button which lets you do data sampling and the sampling will also be used to better understand your data and the model. Uh so I'm just going to show you a screenshot from the uh nextgen

4:32lake like house newsletter but that's what you would get. So you have you know the same place all the column description and you can click on add values and that's going to sample uh the values helping also the like the llm or engineer room to give you better answer. So three main on the metadata and just to give some some insights quickly on this why it's important because sometimes in the value of the column you

4:55can have for example I don't know steps of the sales it can be like prospect then signing like proposing a underscore contract and if you ask a question like hey which steps are we right now it's hard to filter on this specific values so when you capture like the different values within this specific column it will help genie generate better queries

5:17is in the past you had to define this at at the instructions level but what if you have many many columns with many values you won't be spending that amount of time so that's why data sampling can be very important yeah like in all the you know where you have to do queries equal and and all the filters it's it's super useful uh very cool feature um so yeah three main things you want to check

5:39on the metadata you want to make sure you have the description on the tables and the field you want to double check your your primary key and foreign keys and then you also want to make sure you turn on the sampling so that we send some sample example to Jenny and he can better you know answer your questions and apply the filters. If you do that it's already like it's it's going to be a good good first step. I think we still

6:01have more more trips and more tricks and trips. Yeah. You do you want to show us uh what we can do next? Yeah. So the next thing let me share again uh the entire window. Okay. Here we go. The next thing is to define instructions. And I think instructions are very very important. Why? Because it's where you can define

6:24um some spec some specificity to your company. It can be for example uh when does start your fiscal year or maybe what do I mean by air like some definitions or acronyms or maybe uh here for example if I say I am winning a cell what does it mean? which means you need for example to filter by by a closed one or maybe if I have a result please just

6:47show me the first two decimals or maybe avoid showing null values. So basically this is where you can fine-tune the results and you it can be done it uh in iteration mode which means you go ask a question see the results and then if you think that it can be improved you can go and add instructions and you can do it step by step until you until uh you

7:10until you think this is more than good enough to open it to uh to more users. So that's the first step. Yeah. And and you can also track the like once you open it, you can track who is doing what and monitor, you know, all the questions. Exactly. If you see that something is bad like what you see here in the screen, you can check the questions and also you know improve the instruction based on the feedback.

7:32Exactly. But basically you can just go here and see the question that was that was asked. You can either for example if someone is testing your gen room uh this person can leave a thumbs up or maybe a thumbs down. they can show and they can also something import before before showing the code they can also leave some like feedback I think it's a wrong for

7:56example wrong API once I send this basically if I am the person responsible for monitoring the genie room I am I will receive this notification over here I can click and see this person left left a comment I think it's wrong API I can come here and try to fix this issue for example it's not owner ID but I don't know as owner. You can run the

8:18query again and if it's working fine, you can just save it as an instruction which means next time the person ask a similar question or maybe the same question you're sure that you will show the exact result and also as Quinton mentioned in this monitoring UI can filter for example but just by completed and see for example the one uh rating

8:40was negative and just go and correct them and save and save them as interaction. And that's why I mentioned you need to follow an iteration uh to do this an iteration which means but after maybe one week or two week you you know that your genie can be let's say ready to be used by nontechnical users and you can trust the assets you have there. Yeah. And you can so when you click on

9:03add to instruction you would see the queries in the SQL queries menu on the right. Um yeah so that would be all the queries that you add. So every time you have like a very specific question like in in some business I don't know maybe the fiscal quarter means something or the performance means something specific. So you can add the question here and then put the query and the next time someone ask the same you know kind

9:26of question it's going to know how to how to extract the information exactly and you have also something called trusted assets and basically for trusted assets you can define a SQL function in Unity catalog and this function will be brought whenever you're uh you're you're talking about a specific um question. So let me show you an

9:49example here. Here whenever I ask a question what's a pipeline uh what is the pipeline for APAC and MA it will automatically bring this uh SQL query that will run on my data and the response would be something called trusted uh assets. So you will have this this information or that this qu this

10:12answer is trusted. Yeah. So it's very cool if you want to put your key like your main KPI uh like for example really important business metrics uh maybe on you know revenue and stuff like that you can hardcode the query you you have the query it's being reviewed by an analyst so you know the query is correct and every time someone ask a question about the revenue you know you have to is the query and you know you can trust the

10:34answer exactly and I think the next step would be um something called benchmark and I think benchmark are very very important why it's a way where you can add as many questions as you want with their SQL queries and whenever you add new tables or like keep adding more

10:56data you can run those benchmarks automatically and they will give you like for example here I did run those two queries say okay the assessment is good the second one is good which means what what you did was was cool but the second time I did run it because I added some data it needs review which means I may need to go and do run some like I

11:18don't know review the SQL code modify it and save it as an instruction to make sure that this error won't be made next time. So that's what what what we mean by the benchmark and Quinton is there any way to expose Genie outside datab bricks? Yeah, that's a good question and of course the answer is yes, you can use

11:41the new public API. Uh it's super cool. You so you would have Jenny, you can set up the general room like we discussed, you know, all the best practices. You add the command, the key, you make sure you have good instruction, you review the queries, you add your trusted assets, and once you feel confident it's working well, you can just use the API. And the IP is going to be you know just cering the gen room so that you can plug

12:04all the gen capabilities within you know slack teams like a web application anything and then you can just ask question about your business and you're really like the end user you meet the end user where they are so typically like in teams like in the screenshot and I think Databricks in the latest uh blog article they released the stepby-step guide in order to deploy uh

12:26Genie within teams and I find it super cool But you can also deploy it if you want in Databricks apps which can be very useful like if you're using I don't know stream light streamlate application then you just add a chatbot where you can interact and the last thing I wanted to mention is the possibility to drag and drop your CSV files or Excel within Genie. If you want to learn more just

12:49make sure to um to have a look at this recording. Anything else to add? No, that's it. Let's go and let's start creating some nice AI gen room. Exactly. And keep in mind apply the best practices step by step few iterations and after few days your genie will be