Genie Text-to-SQL: Compound AI for Data Exploration

0:00that says hey if we can't even run the sequel uh there's probably it's probably wrong uh so let's try again let's try something different so that JY doesn't startat how much can we actually automate how much can we actually just handle this in the base models help help the like the users have more confidence in the in the answer um there are by the way valid instances where we generate SQL that's wrong but that's because we

0:22didn't actually have enough [Music] context hey folks today we have a superstar we have chia thi senior director engineering of the aibi AKA AI dashboards and aibi Genie and of course we have Hugh with us right there hey

0:45chiao how's going can you introduce yourself and tell us more about you oh boy all right first of all ignore everything you have said about Superstar um nothing I've done has earned any of that praise uh I'm Chia uh I work on aibi uh which is basically data breaks solutions that are trying to make data breakes a lot more accessible to a whole

1:08lot more people hopefully folks who don't even need to have technical sophistication to be able to access and use data Brakes in the ways of dashboards which you all know what dashboards are and Genie which is a way for you to directly talk to the data and get hopefully good answers back um I've been working in this space for quite a few years I used to be at Google

1:30working on product accessors Google analytics basically focusing on helping marketers and advertisers uh talk to their data in effect and get understand their data and uh a couple years ago I was talking to some of the folks from Databricks and basically came with this idea where we could try to expand this to a lot more than just marketers and advertisers let's actually try to make data accessible to you know every single

1:54function every single company out there and so a lot of the ideas behind aibi came out of that and trust me Jin is really simplifying how people are accessing and asking questions and having F like Fast answers to every single question they're asking specifically like the easy ones that goes like hey tell me more about how

2:16many sales it I did last week how many failures we there how many tickets were opened do me the aggregation add some specific details we'll Deep dive more on this little bit later so I think I announced aibi at D bricks and isit unseen can you I mean before the bricks we had dashboards I think we had dashboards V2 leg views and now we have aibi

2:40dashboard Leg view can you can you tell us what's going on here yeah so aibi dashboards is just a rebranding of l v dashboards they're the same thing um prior to that we had uh sort of some dashboarding functionality inside of data break SQL already um and some of that came out of an earlier sort of redash acquisition uh and uh sort of the revamp and sort of the rebuild

3:03within that into Data breaks um we basically looked at that and said okay currently dashboards is sort of a feature of data warehousing but we'd like to actually make it its own first class citizen so that I can actually focus more on sort of expanding this capability known as business intelligence to as many people as possible to make it as useful to as many

3:26companies as possible and so that's where we basically refactored dashboards out of some of the data warehousing stuff and started focusing on it as its own Standalone effort uh and then along the way uh coincidentally sort of uh the person who started redh aric um was uh doing a hackathon with another engineer Simon who uh basically around this

3:49concept of hey data braks is still sort of a little too hard to use in getting access to its data for any person um and so let's actually try to see if we can rethink things from scratch and the project that eventually became Genie came out of that so so if I recap we so there bricks had no Nobi no dashboards and then we had the acquisition of free Dash when was it like two two three

4:12years a couple years ago yeah2 I guess if I'm not 2001 it's three years ago now yeah three years ago and then we so we did the first dashboard based on redash and then we did an update on that uh CLE view so that it's much easier to create the queries and manage everything but yet it wasn't enough and I feel like the all the AI thing was missing and that's where the aibi dashboard comes in

4:35yeah so so I would say Genie is sort of a Confluence of a couple of different efforts um one being this hackathon I mentioned where we started exploring this idea of what is the right interface for people to get access to their data much more easily ideally talk to their data since at the time geni was just starting to become a lot really popular overnight um and then the other was uh

4:57so the underlying research around things like text to SQL which some of our ml teams started doing in parallel and saying how much can we actually automate how much can we actually just handle this in the base models versus how much do we actually need to sort of talk to users and get more context from them and just to clarify for the folks who are still using um data break seal the Legacy dashboards they can migrate them easily

5:20to Lake View with a button on the top and make sure to do it that's right there's a clone button we try to make it as easy as possible but if you feel like it's still too hard give us feedback and uh Chia you you told that you told us that J started with hackathon so can you tell us more about the steps that we follow I mean for my case I only was

5:42able to track the name the rebranding we start with called project 1B then was called uh data rooms then it switched to Genie yeah we renamed it a fair bit uh honestly the name was sort of many in many times secondary to the effort itself s which was really trying to say okay 1B the original name of it

6:04was basically saying let's see how how quickly we can actually get this to a billion users right that was what 1B was for and uh a lot of the goal and the spirit behind that was let's make a version of data breaks that's as simple for people to use as possible so that as many people can use it as possible and so the idea of it was okay what's simpler than just you know talking and asking English natural language

6:27questions or any language natural language questions and and getting answers back accurately of course this is a really hard Challenge on many layers um starting with how do you actually build the right models to actually understand your data correctly and answer the questions as accurately as possible and that's assuming we have all the context but we don't so how do you actually build the right interfaces so that we can get context from people

6:51uh as effectively as possible make that work as easy as possible so there's a lot of interesting challenges that come into Genie but the focus originally with the hackathon was okay let's actually get the interface right for the person who's ultimately trying to make data driven decisions let's actually try to make sure that it's possible for folks to be able to sort of ask questions get

7:13answers back trust those answers potentially verify if they have the technical sophistication to do so and sort of make sure that that workflow and that feedback loop is as sort of clean as possible and then of course you got to solve all the hard problems around it like how do you actually make sure that the answer is uh right as often as possible so for everybody Jenny is the place where you can go and you ask uh

7:36like plain English question like hey can you give me an Insight on something and then you have all the magic happening behind the scen so dat bricks kind of knowing your data understanding your question is going to write down a SQL query generate the visualization and give you back the answer so that any you know business user can do can get an Insight basically I think a good flow what I like to show when I do that is

7:59you know typically you would have a leg view dashboard so like it's a dashboard that you share like the bi Team Real and you share the dashboard with your business team and then from the dashboard you would see that maybe something is wrong you know something is triggering your attention you know there is something you need to deal with uh but to F the answer is not directly in the dashboard so before that you will need to go back ask you know the analyst

8:21hey can you give me some insight and extract on that and that and that which is really slow and you know it can take a couple of days or weeks and now the business user can just you know open the geny room from the dashboard and say Hey you know can you give me more details on whatever they want to know and then the engine is going to them the magic behind can you tell us more about like how is it working how do you because

8:43it's really working well and I know there is a lot of secret Source behind but can you can you yeah try to go in the details and tell us how you um we debated a lot honestly on like if you have gen spaces do we even still need dashboards for example and uh we basically settled on yes uh dashboards are there for the top 10% of sort of recurring every day you always want to

9:06kind of check on this kind of questions and Genie for the digging deeper the bottom 90% if you will of questions that a dashboard will not be able to answer because sort of you're not going to have the data analysts attention to be able to build every single potential sort of a slicing of a particular uh

9:28dashboard to be able to answer every single fine gr query um and so we thought of Genie ultimately as that kind of compliment and so the way we built it is in some ways reflective of that um it starts out with you know more of a general purpose model uh your uh sort of the general ability to answer a sort of

9:50textas SQL standard questions across all customers in a standard way and that's something that to be fair we haven't perfected yet right we're improving all the time and it's able to get good answers on many classes of queries it still fumbles on some classes and we're working on closing those and a lot of the so a lot of the work here is actually in working with customers

10:12getting their feedback on what classes of queries are most important for them uh and then making sure that we're able to prioritize sort of improving those and making those as accurate as possible so that's the base model to clarify for when you say base model it could be for example I know lat three that we find tune to do Tex squel or something like that yeah in in some ways um so the the way we've sort of framed our

10:34solution is this notion of sort of a compound AI system and the reason we did so is because the individual models involved in some ways are slightly secondary to the fact that it's a composite of several different models each solving a particular sub problem for example the sub problem of generating SQL the sub problem of

10:56explaining what the SQL is doing the sub problem of figuring out what are valid table values column values that are even eligible to be put into the we Clause the sub problem of suggesting what our potential interesting question for you to ask all of those each of those could in theory be powered by a different model um and that's okay because uh we're looking for basically best of breed across all these models some of

11:20them may be fine some of them PRP engineered Etc um in order to actually deliver the most accurate answers and the best experience for every customer um and so in that regard the most important piece of this that is often easy to uh Overlook is the feedback loop itself is how we actually verify given a

11:42particular composition of models is it actually doing better than the last thing we were doing and is it actually doing good enough for all the customers most of the customer consider this is especially hard because you know we actually should not be and we will not touch any customers data or the questions that they're asking without their permission and by default we do not do so which means that we do not

12:05actually directly learn whenever they get an answer right or get an answer wrong most of the time we only learn when they complain to us and say hey this is actually wrong uh and so the feedback loop is actually fairly lossy in that regard so we've had to do a lot of interesting sort of Innovations on the data that we do have on whether it's synthetic data or whether it's data

12:28that's use within data brakes itself or various other things um in order to have a tight enough feedback loop to be able to learn and iterate enough on the models Capon systems and multiple sub model trying you know to do their best to well to provide something at each step of the literally that's yeah and like could so typical

12:52flow would be hey you know based on I guess you know first of all get give me the schema we get the schema based on the schema can you try to write the to write the SQL then I imagine we will try to run the SQL if it's failing we try to fix it and and some kind of chain of in in very early iteration Vini right it would actually generate SQL that was not

13:14even runable by data brakes itself so that's a very clear feedback loop for us that says hey if we can't even run the SQL uh there's probably it's probably wrong uh so let's try again let's try something different um there are by the way valid instances where we generate SQL that's wrong but that's because we didn't actually have enough context right like for example we may generate a column value that doesn't even exist in

13:36the database but then we don't have enough context around is that because simply in the range of data that we're looking at that value just didn't occur or is that actually because that value is completely invalid hallucinated made up by someone uh and so that's where additional context in the form of things like instructions in the form of things like column comments Etc is super

13:58helpful to actually let us sort of constrain the search bace a little bit Yeah and since you talk about uh comments on columns we know that Unity catalog made this possible because it because we have some metadata link to every single table comments on the table comments for every single columns so how did uh UC make uh or help Genie becoming

14:22a reality yeah um UC I think is a necessary building block to something like Genie um in the sense that you need a source of Truth for not just governance but semantics and UC is increasingly through starting with things like comons going from just a place where you declare hey

14:46this is who can access what to hey this is actually what my data means and if you think about it Genie is built off like I mentioned that general purpose model um that is uh the same across all customers but then the secret sauce on top of it is actually we're trying to learn on your data as a customer we're trying to refine Genie to be most useful

15:08for your data and that can only happen if you have the right context provided in places like you see for it otherwise we're just guessing and you just mentioned security I imagine you know I mean I would be concerned about security like when you run the query you have to make sure that Jenny runs the query only on what the current user can see and you don't lick any anything is that a big challenge yeah so the nice thing is

15:31Genie doesn't actually have to worry about a lot of that itself because it's largely just generating the queries but then it's talking to existing data Brak services like for example the SQL execution API it's running the queries through those so all the governance around who can see the queries who can see the results Etc is already handled by the data Bri SQL layers and Genie

15:53sort of just interacting with it as a client application very similar to say a third party bi tool building built on top of us yeah I guess you just have to make sure it's a readly system so that Jen doesn't start to mess up your later that's right it is yeah both Genie and actually uh aibi dashboards are set up to be readon uh we get uh bugs uh or F

16:15filed complaints at times from folks who are like hey I want to actually use my dashboards or Genie to you know alter my database in a certain way and we have to explain to him no this is why that possible and feasible but we're not going to build it sorry and just like we know that Jor now relies on as I mentioned comments and uh

16:39comments over columns and table are we uh thinking about adding for example access to the query history so we can understand how many times users are querying this table maybe also the relevant details because now if you go to table insights for every single table you're going to see the users who query this table most the most the the queries you run most on this table and also the tables you join the most with

17:03this one so yeah is J able is able to access all those details or it's on on in the road map I would say it's something we're actively researching um query history lineage any of the things that give us more context about not just what the what what the meaning of the data is but also sort of who thinks the

17:25data means what and who can be trusted um over time we have to infer a whole lot of that and then the way we'll probably approach it is to try to infer as much as possible but then also try to suggest some of it to the folks who are sort of curating the genie spaces not necessarily just the people who are using it but the people who are curating it and say hey we're inferring that this

17:48might be what the data means we're inferring that this might actually be um the so the semantics behind this particular column or this particular table are we right about that right so like some of this might actually go back into table columns and common column columns as AI suggestions uh so that we actually capture that and have the validation Loop there some of it will go directly into Genie because we can

18:10actually just directly use it so we'll probably do some of each of those things because ultimately we're trying to make sure that this doesn't just serve a single user or a single group of users but it serves the whole organization and it has the right governance and the right sort of uh validation built in so that everyone can actually check for is this right or not I feel like on this system there is a big question and maybe

18:33issue on trust uh because if if I ask a question and the system is giving me back you know something if I am to take action Bas on the inside I really need to trust uh the actual results and like one way to trust it is to actually see the SQL query and make sure it's okay uh but typically business analyst won't

18:55know exactly if it's trustable so what's your thinking about that and how can we solve it yeah no this is one of the hardest problems in for something like Genie which is how do you actually establish trust and also not uh create overconfidence for uh for everyone who's using it both the technical and the business users for the technical users as you mentioned it's a little easier because if they can speak SQL they can

19:18validate SQL that's great for the business users things we've tried to do in this regard are things like for example um trying to give a sort of a natural language explanation of what the SQL is doing so they have a way of sort of Sanity checking is that actually headed in the right direction uh and we're also exploring the sort of uh Notions of sort of certified or validated functions certified or

19:40validated sort of queries answers Etc as a way of basically saying hey uh you may not be able to understand what this is doing but you can trust the author of this sort of thing uh that uh that we're using you can sort of trust the author of The Tool rather than know the internals of the tools itself so it becomes more Trust trusting the people who created uh and validated uh these

20:03answers rather than trusting the answer itself and if you think about it this is how a dashboard works today right like you as if you're using a dashboard you're not looking at the underlying SQL of fair bit of the instances you're sort of trusting that the author is competent and has generated the right things yeah so we could have a system where uh like SQL experts kind of build metrics and save the metrics and explain what each

20:26metric is doing uh and then the end like the AI would be just in charge of kind of composing the queries based on the metric and and I mean we have to be careful not to make that too rigid because then it sort of just becomes a very expensive dashboard um but uh so so a lot of this is in trying to make sure that we have the flexibility of natural language and gen but then also have sort

20:49of trust where it matters and some of this can be a two-way conversation where you know if the business user is looking at it and saying I don't know if I could trust it we want to facilitate workflows where they can then go ask the Analyst at that point hey this is business critical I'm about to go present to the SE Suite about this can you please double check and make sure and then if the analyst blesses this it's not just that that business user

21:12gets the benefit everyone else who in the future use ask the exact same question we also get those benefits yeah yeah I mean that make lots of sense I could see that I can ask any question I'm not sure hey you know can you double check and then it becomes the new metrics and then you can have like a yeah and I think also you can improve the uh can improve the genie so what you can think about like create for example

21:34a genie for each I don't know business team I don't know marketing team uh sales team you can just add the relevant tables with the right comments and then you can add like good instructions so this is what I mean by this product those are the relevant ones uh that's what I mean if I have value one in my comment this means this is I don't know the best selling uh product and then you

21:58can what I what I thought about last time that was go check the query history and see the most the queries that I have the most like uh had on this specific SQL warehouse and copy them in the instruction with the explanation and this help help help the like the users have more confidence in the in the answer and by time like like now now

22:20they using Genie more they have this trust on J to have like good results that's right yeah I mean query history is a great source of sort of what have what questions have people been asking before now today what we lack is the mapping of given a query in the query history what was the English or natural language intents behind that question

22:42and so uh that's the those are the semantics that we're trying to encourage folks to sort of provide a lot a lot more through genie in as easy a way as possible can can you R thats like can you have a model based on a query kind of generating the English we do actually today already right if you if you use Genie today um every time we generate a query we also generate an English description with it and so uh

23:04part of uh the challenge is making sure that that's as accurate as possible as well and I think it will be um I don't know I'm just thinking about maybe probably in the road map you can of have a genie like link to the system table that can be accessed only for the user where he can see the question he asked and the SQL that was generated it's like in inference table when using the llms

23:28so at the end of the month an analyst can just go and check yeah something like so the nice thing is um we don't quite have this in the system table yet but we do already have this in the UI there is a monitoring tab where you can go and look at all the uh questions that's been asked for this particular gen space if you're an author of that space and you can also look at whether people like the answers like there's

23:50thumbs up thumbs down that is baked in every question and uh you can actually filter by which questions got a thumbs up which questions got a thumbs down what was the SQL generated what was the chat history of context behind that so all of that is actually available today for people who are trying to cure these gen spaces I think you built one of the most advanced you know compound AI

24:12system out there can you can you share any best practices with our customers we would like to do the same kind of system maybe not doing you know a joury but something they are working on for their own business the secret I don't know if the most advanced one but it is certainly a pretty complex one um I think with any AI system especially given all the sort of the problemistic

24:34nature of the behavior there um the most important thing is to figure out the feedback loop as early as possible and figure out how you can actually make sure the system is improving over time um and what signals do you have to actually confirm that ideally as close to the customer experience as possible uh and so looking at the n2n in in for

24:56the for example for Genie the n2n sort of request response is a natural language question with the data and the SQL that's coming back and so we then do a lot of work to figure out okay how do we actually convince ourselves that we have a representative sample of such natural language questions to SQL into the data such that every time we make a model change every time we even make an

25:18interface change Etc we can actually confirm that it's headed in the right direction so any other compound AI system that tends to be one of the hardest parts actually getting that contract right up front and getting enough samples of real customer interactions based on that contract where you would expect that to be correct so the golden data set if you will yeah makes sense and can you share

25:42maybe a bit on the eval strategy that you have because I mean if you generate SQL you can get the same results with many different SQL so how do you know it's working well at the end yeah and I mean it's a I would say the eval strategy is always a best for work in progress right and some of uh we came up with a fairly extensive sort of set of what we believe to be representative

26:04at the beginning um and um some of that is informed by our own experience some of that is informed by looking at what folks internally are doing at data breaks as they're playing around with Genie um but then also over time a lot of is going to be informed by customer feedback like if you give us feedback and if you give us enough permission to uh actually tell what you're trying to

26:26do in that feedback um then we learn from that and then we will sort of incorporate it that over time to make sure that you know we don't trip over the same thing twice that makes sense and I thought about something it's um yeah it's it's a hard question so do you know why a data analyst is always a

26:48good dancer oh boy is this a joke I I'm terrible at these because they know how to Pivot oh no my means to remove your faiv table access we might cut that one [Laughter]

27:12youself I walk right into that one and and and by the way can you tell us more about the road map like yeah what may be coming to jinny there's a lot I can't talk about yet uh but I would say uh the things that we've already started hinting at even at Summit are things like for example tighter integration

27:35between dashboards and Genie where we can try to use a dashboard to sort of bootstrap a genie space so that you have to do less work in setting it up ideally if you can ask a dashboard a question you should be able to ask the exact same question Genie and then you can ask all the questions around it so that's a sort of a big area that we are sort of increasingly focusing on and you know some of us just making sure that as easy

27:58to use as possible um and then sort of making Genie accessible to as many people as possible like we know a lot of folks uh don't actually want uh any random business user to be in their workspaces uh we want we know that they want business users to maybe just have access to Genie and nothing else and so we're we're looking at various ways of making that um much more uh usable as

28:21well um and so those are I would say two of the bigger themes but then a big part of the road map that you may not see as obvious sort of big launch features are just like refining the quality of the answers and making sure that we get the answers right by default a lot more frequently so that you as a author have to provide fewer and fewer instructions over time and so uh hopefully that will

28:43you will just see over time as s of natural improvements in Genie ni and and and any so what's the integration between genie in the AI assistant that we have like if you ask a business question in the AI assistant for example from notebook is it going to trigger Jenny also how how does it work oh that's interesting so you're thinking assistant specifically yeah yeah so so

29:06we basically framed assistant as more of the companion AI thing for technical users and Genie as the thing that answers your questions for business users and so in some ways uh they're complimentary to each other for two different crowds um I could certainly see over time things that you learn from Genie or things things that you learn from assistant benefit in each other but

29:29we'll have to be very careful about so the governance angle making sure that the right people can only answer the right uh access to answers for the right questions um given the level of access they have in unity calog and so again the nice thing about Unity catalog is of it will solve all of the who gets access to what kind of questions so that Genie and assistant can sort of act as purely

29:52applications on top of that yeah then and I I think like like uh for those who are watching this recording after 30 minutes you know why chiao and his team are superstars they are really delivering a master piece and working on one of the coolest projects

30:14ever and because like it's easy to make technical stuff but not democratizing access and makes making the data accessible to everyone so really thank you for your hard work uh thank you I'm going to go hide behind that chair or something because no the credit really goes to the team there's a lot of really cool really Innovative

30:36work happening across the board whether it's building the right models whether it's scaling a lot of really powerful systems whether it's figuring out the right human computer interaction so that we can actually make the work of both the analysts and the business user as easy as possible there's a lot of fun things we're doing here and uh we're nowhere near done yet so uh looking forward all the stuff that we're doing

30:59in the coming quarters and years all right thank you so much yeah let's give it to try you know like to day breaks go in Jenny start your first discussion and and you know all all the magic happening for you yeah and tell us how you like it uh we always love more feedback that's how we actually grow the product faster perfect awesome thank you CH thank you thank you