Executive Interview
Text-to-SQL with Databricks AIBI Genie with Chao Cai
Chao Cai · Sr Director Engineering · LinkedIn
that says hey if we can't even run the sequel uh there's probably it's probably wrong uh so let's try again let's try something different so that JY doesn't startat how much can we actually
automate how much can we actually just handle this in the base models help help the like the users have more confidence in the in the answer um there are by the way valid instances where we generate
SQL that's wrong but that's because we didn't actually have enough [Music] context hey folks today we have a superstar we have chia thi senior director engineering of the aibi AKA AI
dashboards and aibi Genie and of course we have Hugh with us right there hey chiao how's going can you introduce yourself and tell us more about you oh boy all right first of all ignore
everything you have said about Superstar um nothing I've done has earned any of that praise uh I'm Chia uh I work on aibi uh which is basically data breaks solutions that are trying to make data
breakes a lot more accessible to a whole lot more people hopefully folks who don't even need to have technical sophistication to be able to access and use data Brakes in the ways of
dashboards which you all know what dashboards are and Genie which is a way for you to directly talk to the data and get hopefully good answers back um I've been working in this space for quite a
few years I used to be at Google working on product accessors Google analytics basically focusing on helping marketers and advertisers uh talk to their data in effect and get understand
their data and uh a couple years ago I was talking to some of the folks from data bricks and basically came with this idea where we could try to expand this to a lot more than just marketers and
advertisers let's actually try to make data accessible to you know every single function every single company out there and so a lot of the ideas behind aibi came out of that and trust me Jin is really simplifying
how people are accessing and asking questions and having F like Fast answers to every single question they're asking specifically like the the easy ones that goes like hey tell me more about how
many sales it I did last week how many failures we there how many tickets were opened do me the aggregation add some specific details we'll Deep dive more on this little bit later so I think I
announced aibi at D bricks and isit unseen can you I mean before the bricks we had dashboards I think we had dashboards V2 leg views and now we have aibi dashboard Leg view can you can you tell
us what's going on here yeah so aibi dashboards is just a rebranding of l v dashboards they're the same thing um prior to that we had uh sort of some dashboarding functionality inside of
data break SQL already um and some of that came out of an earlier sort of redash acquisition uh and uh sort of the the revamp and sort of the rebuild within that into Data breaks um we
basically looked at that and said okay currently dashboards is sort of a feature of data warehousing but we'd like to actually make it its own first class citizen so that I can actually
focus more on sort of expanding this capability known as business intelligence to as many people as possible to make it as useful to as many companies as possible and so that's where we basically refactored dashboards
out of some of the data warehousing stuff and started focusing on it as its own Standalone effort uh and then along the way uh coincidentally sort of uh the person who started redh aric um was uh
doing a hackathon with another engineer Simon who uh basically around this concept of hey data braks is still sort of a little too hard to use in getting access to its data for any person um and
so let's actually try to see if we can rethink things from scratch and the project that eventually became Genie came out of that so so if I recap we so there bricks had no Nobi no dashboards
and then we had the acquisition of free Dash when was it like two two three years a couple years ago yeah2 I guess if I'm not 2001 it's three years ago now yeah three years ago and then we so we
did the first dashboard based on redash and then we did an update on that uh CLE view so that it's much easier to create the queries and manage everything but yet it wasn't enough and I feel like the
all the AI thing was missing and that's where the the aibi dashboard comes in yeah so so I would say Genie is sort of a Confluence of a couple of different efforts um one being this hackathon I
mentioned where we started exploring this idea of what is the right interface for people to get access to their data much more easily ideally talk to their data since at the time geni was just
starting to become a lot really popular overnight um and then the other was uh so the underlying research around things like text to SQL which some of our ml teams started doing in parallel and
saying how much can we actually automate how much can we actually just handle this in the base models versus how much do we actually need to sort of talk to users and get more context from them and
just to clarify for the folks who are still using um data break seal the Legacy dashboards they can migrate them easily to Lake View with a button on the top and make sure to do it that's right
there's a clone button we try to make it as easy as possible but if you feel like it's still too hard give us feedback and uh Chia you you told that you told us that J started with hackathon so can you
tell us more about the steps that we follow I mean for my case I only was able to track the the name the rebranding we start with called project 1B then was called uh data rooms then it
switched to Genie yeah we we renamed it a fair bit uh honestly the name was sort of many in many times secondary to the effort itself s which was really trying to say okay 1B the original name of it
was basically saying let's see how how quickly we can actually get this to a billion users right that was what 1B was for and uh a lot of the goal and the spirit behind that was let's make a
version of data breaks that's as simple for people to use as possible so that as many people can use it as possible and so the idea of it was okay what's simpler than just you know talking and
asking English natural language questions or any language natural language questions and and getting answers back accurately of course this is a really hard Challenge on many layers um starting with how do you
actually build the right models to actually understand your data correctly and answer the questions as accurately as possible and that's assuming we have all the context but we don't so how do
you actually build the right interfaces so that we can get context from people uh as effectively as possible make that work as easy as possible so there's a lot of interesting challenges that come
into Genie but the focus originally with the hackathon was okay let's actually get the interface right for the person who's ultimately trying to make data driven decisions let's actually try to
make sure that it's possible for folks to be able to sort of ask questions get answers back trust those answers potentially verify if they have the technical sophistication to do so and
sort of make sure that that workflow and that feedback loop is as sort of clean as possible and then of course you got to solve all the hard problems around it like how do you actually make sure that
the answer is uh right as often as possible so for everybody Jenny is the place where you can go and you ask uh like plain English question like hey can you give me an Insight on something and
then you have all the magic happening behind the scen so dat bricks kind of knowing your data understanding your question is going to write down a SQL query generate the visualization and
give you back the answer so that any you know business user can do can get an Insight basically I think a good flow what I like to show when I do that is you know typically you would have a leg
view dashboard so like it's a dashboard that you share like the bi Team Real and you share the dashboard with your business team and then from the dashboard you would see that maybe
something is wrong you know something is triggering your attention you know there is something you need to deal with uh but to F the answer is not directly in the dashboard so before that you will
need to go back ask you know the analyst hey can you give me some insight and extract on that and that and that which is really slow and you know it can take a couple of days or weeks and now the
business user can just you know open the geny room from the dashboard and say Hey you know can you give me more details on whatever they want to know and then the engine is going to them the the magic
behind can you tell us more about like how is it working how do you because it's really working well and I know there is a lot of secret Source behind but can you can you yeah try to go in
the details and tell us how you um we debated a lot honestly on like if you have gen spaces do we even still need dashboards for example and uh we basically settled on yes uh dashboards
are there for the top 10% of sort of recurring every day you always want to kind of check on this kind of questions and Genie for the the digging deeper the the bottom 90% if you will of questions
that a dashboard will not be able to answer because sort of you're not going to have the data analysts attention to be able to build every single potential sort of a slicing of a particular uh
dashboard to be able to answer every single fine gr query um and so we we thought of Genie ultimately as that kind of compliment and so the way we built it is in some ways reflective of that um it
starts out with you know more of a general purpose model uh your uh sort of the general ability to answer a sort of textas SQL standard questions across all customers in a standard way and that's
something that to be fair we haven't perfected yet right we're improving all the time and it's able to get good answers on many classes of queries it still fumbles on some classes and we're
working on closing those and a lot of the so a lot of the work here is actually in working with customers getting their feedback on what classes of queries are most important for them
uh and then making sure that we're able to prioritize sort of improving those and making those as accurate as possible so that's the base model to clarify for when you say base model it could be for
example I know lat three that we find tune to do Tex squel or something like that yeah in in some ways um so the the the way we've sort of framed our solution is this notion of sort of a
compound AI system and the reason we did so is because the individual models involved in some ways are slightly secondary to the fact that it's a composite of several different models
each solving a particular sub problem for example the sub problem of generating SQL the sub problem of explaining what the SQL is doing the sub problem of figuring out what are valid
table values column values that are even eligible to be put into the we Clause the sub problem of suggesting what our potential interesting question for you to ask all of those each of those could
in theory be powered by a different model um and that's okay because uh we're looking for basically best of breed across all these models some of them may be fine some of them PRP engineered Etc um in order to actually
deliver the most accurate answers and the best experience for every customer um and so in that regard the most important piece of this that is often easy to uh Overlook is the feedback loop
itself is how we actually verify given a particular composition of models is it actually doing better than the last thing we were doing and is it actually doing good enough for all the customers
most of the customer consider this is especially hard because you know we actually should not be and we will not touch any customers data or the questions that they're asking without
their permission and by default we do not do so which means that we do not actually directly learn whenever they get an answer right or get an answer wrong most of the time we only learn
when they complain to us and say hey this is actually wrong uh and so the feedback loop is actually fairly lossy in that regard so we've had to do a lot of interesting sort of Innovations on
the data that we do have on whether it's synthetic data or whether it's data that's use within data brakes itself or various other things um in order to have a tight enough feedback loop to be able
to learn and iterate enough on the models Capon systems and multiple sub model trying you know to do their best to well to provide something at each step of the literally that's yeah and like could so typical
flow would be hey you know based on I guess you know first of all get give me the schema we get the schema based on the schema can you try to write the to write the SQL then I imagine we will try
to run the SQL if it's failing we try to fix it and and some kind of chain of in in very early iteration Vini right it would actually generate SQL that was not even runable by data brakes itself so
that's a very clear feedback loop for us that says hey if we can't even run the SQL uh there's probably it's probably wrong uh so let's try again let's try something different um there are by the
way valid instances where we generate SQL that's wrong but that's because we didn't actually have enough context right like for example we may generate a column value that doesn't even exist in
the database but then we don't have enough context around is that because simply in the range of data that we're looking at that value just didn't occur or is that actually because that value
is completely invalid hallucinated made up by someone uh and so that's where additional context in the form of things like instructions in the form of things like column comments Etc is super
helpful to actually let us sort of constrain the search bace a little bit Yeah and since you talk about uh comments on columns we know that Unity catalog made this possible because it
because we have some metadata link to every single table comments on the table comments for every single columns so how did uh UC make uh or help Genie becoming a reality yeah um UC I think is a
necessary building block to something like Genie um in the sense that you need a source of Truth for not just governance but semantics and UC is increasingly through starting with things like comons going
from just a place where you declare hey this is who can access what to hey this is actually what my data means and if you think about it Genie is built off like I mentioned that general purpose
model um that is uh the same across all customers but then the secret sauce on top of it is actually we're trying to learn on your data as a customer we're trying to refine Genie to be most useful
for your data and that can only happen if you have the right context provided in places like you see for it otherwise we're just guessing and you just mentioned security I imagine you know I
mean I would be concerned about security like when you run the query you have to make sure that Jenny runs the query only on what the current user can see and you don't lick any anything is that a big
challenge yeah so the nice thing is Genie doesn't actually have to worry about a lot of that itself because it's largely just generating the queries but then it's talking to existing data Brak
services like for example the SQL execution API it's running the queries through those so all the governance around who can see the queries who can see the results Etc is already handled
by the data Bri SQL layers and Genie sort of just interacting with it as a client application very similar to say a third party bi tool building built on top of us yeah I guess you just have to
make sure it's a readly system so that Jen doesn't start to mess up your later that's right it is yeah both Genie and actually uh aibi dashboards are set up to be readon uh we get uh bugs uh or F
filed complaints at times from folks who are like hey I want to actually use my dashboards or Genie to you know alter my database in a certain way and we have to explain to him no this is why that
possible and feasible but we're not going to build it sorry and just like we know that Jor now relies on as I mentioned comments and uh comments over columns and table are we uh thinking about adding for example
access to the query history so we can understand how many times users are querying this table maybe also the relevant details because now if you go to table insights for every single table
you're going to see the users who query this table most the most the the the queries you run most on this table and also the tables you join the most with this one so yeah is J able is able to
access all those details or it's on on in the road map I would say it's something we're actively researching um query history lineage any of the things that give us more context about not just
what the what what the meaning of the data is but also sort of who thinks the data means what and who can be trusted um over time we we have to infer a whole lot of that and then the the way we'll
probably approach it is to try to infer as much as possible but then also try to suggest some of it to the folks who are sort of curating the genie spaces not necessarily just the people who are
using it but the people who are curating it and say hey we're inferring that this might be what the data means we're inferring that this might actually be um the so the semantics behind this
particular column or this particular table are we right about that right so like some of this might actually go back into table columns and common column columns as AI suggestions uh so that we
actually capture that and have the validation Loop there some of it will go directly into Genie because we can actually just directly use it so we'll probably do some of each of those things
because ultimately we're trying to make sure that this doesn't just serve a single user or a single group of users but it serves the whole organization and it has the right governance and the
right sort of uh validation built in so that everyone can actually check for is this right or not I feel like on this system there is a big question and maybe issue on trust uh because if if I ask a
question and the the system is giving me back you know something if I am to take action Bas on the inside I really need to trust uh the actual results and like one way to trust it is to actually see
the SQL query and make sure it's okay uh but typically business analyst won't know exactly if it's trustable so what's your thinking about that and how can we solve it yeah no this is one of the
hardest problems in for something like Genie which is how do you actually establish trust and also not uh create overconfidence for uh for everyone who's using it both the technical and the
business users for the technical users as you mentioned it's a little easier because if they can speak SQL they can validate SQL that's great for the business users things we've tried to do
in this regard are things like for example um trying to give a sort of a natural language explanation of what the the SQL is doing so they have a way of sort of Sanity checking is that actually
headed in the right direction uh and we're also exploring the sort of uh Notions of sort of certified or validated functions certified or validated sort of queries answers Etc as a way of basically saying hey uh you may
not be able to understand what this is doing but you can trust the author of this sort of thing uh that uh that we're using you can sort of trust the author of The Tool rather than know the
internals of the tools itself so it becomes more Trust trusting the people who created uh and validated uh these answers rather than trusting the answer itself and if you think about it this is
how a dashboard works today right like you as if you're using a dashboard you're not looking at the underlying SQL of fair bit of the instances you're sort of trusting that the author is competent
and has generated the right things yeah so we could have a system where uh like SQL experts kind of build metrics and save the metrics and explain what each metric is doing uh and then the end like
the AI would be just in charge of kind of composing the queries based on the metric and and I mean we have to be careful not to make that too rigid because then it sort of just becomes a
very expensive dashboard um but uh so so a lot of this is in trying to make sure that we have the flexibility of natural language and gen but then also have sort of trust where it matters and some of
this can be a two-way conversation where you know if the business user is looking at it and saying I don't know if I could trust it we want to to facilitate workflows where they can then go ask the
Analyst at that point hey this is business critical I'm about to go present to the SE Suite about this can you please double check and make sure and then if the analyst blesses this
it's not just that that business user gets the benefit everyone else who in the future use ask the exact same question we also get those benefits yeah yeah I mean that make lots of sense I
could see that I can ask any question I'm not sure hey you know can you double check and then it becomes the new metrics and then you can have like a yeah and I think also you can improve
the uh can improve the genie so what you can think about like create for example a genie for each I don't know business team I don't know marketing team uh sales team you can just add the relevant
tables with the right comments and then you can add like good instructions so this is what I mean by this product those are the relevant ones uh that's what I mean if I have value one in my
comment this means this is I don't know the best selling uh product and then you can what I what I thought about last time that was go check the query history and see the most the queries that I have
the most like uh had on this specific SQL warehouse and copy them in the instruction with the explanation and this help help help the like the users have more confidence in the in the
answer and by time like like now now they using Genie more they have this trust on J to have like good results that's right yeah I mean query history is a great source of sort of what have
what questions have people been asking before now today what we lack is the mapping of given a query in the query history what was the English or natural language intents behind that question
and so uh that's the those are the semantics that we're trying to encourage folks to sort of provide a lot a lot more through genie in as easy a way as possible can can you R thats like can
you have a model based on a query kind of generating the the English we do actually today already right if you if you use Genie today um every time we generate a query we also generate an
English description with it and so uh part of uh the challenge is making sure that that's as accurate as possible as well and I think it will be um I don't know I'm just thinking about maybe
probably in the road map you can of have a genie like link to the system table that can be accessed only for the user where he can see the question he asked and the SQL that was generated it's like
in inference table when using the llms so at the end of the month an analyst can just go and check yeah something like so the nice thing is um we don't quite have this in the system table yet
but we do already have this in the UI there is a monitoring tab where you can go and look at all the uh questions that's been asked for this particular gen space if you're an author of that
space and you can also look at whether people like the answers like there's thumbs up thumbs down that is baked in every question and uh you can actually filter by which questions got a thumbs
up which questions got a thumbs down what was the SQL generated what was the chat history of context behind that so all of that is actually available today for people who are trying to cure these
gen spaces I think you built one of the most advanced you know compound AI system out there can you can you share any best practices with our customers we would like to do the same kind of system
maybe not doing you know a joury but something they are working on for their own business the secret I don't know if the most advanced one but it is certainly a pretty complex one um
I think with any AI system especially given all the sort of the problemistic nature of the behavior there um the most important thing is to figure out the feedback loop as early as possible and
figure out how you can actually make sure the system is improving over time um and what signals do you have to actually confirm that ideally as close to the customer experience as possible
uh and so looking at the n2n in in for the for example for Genie the n2n sort of request response is a natural language question with the data and the SQL that's coming back and so we then do
a lot of work to figure out okay how do we actually convince ourselves that we have a representative sample of such natural language questions to SQL into the data such that every time we make a
model change every time we even make an interface change Etc we can actually confirm that it's headed in the right direction so any other compound AI system that tends to be one of the
hardest parts actually getting that contract right up front and getting enough samples of real customer interactions based on that contract where you would expect that to be correct so the golden data set if you
will yeah makes sense and can you share maybe a bit on the eval strategy that you have because I mean if you generate SQL you can get the same results with many different SQL so how do you know
it's working well at the end yeah and I mean it's a I would say the eval strategy is always a best for work in progress right and some of uh we we came up with a fairly extensive sort of set
of what we believe to be representative at the beginning um and um some of that is informed by our own experience some of that is informed by looking at what folks internally are doing at data
breaks as they're playing around with Genie um but then also over time a lot of is going to be informed by customer feedback like if you give us feedback and if you give us enough permission to
uh actually tell what you're trying to do in that feedback um then we learn from that and then we will sort of incorporate it that over time to make sure that you know we don't trip over
the same thing twice that makes sense and I thought about something it's um yeah it's it's a hard question so do you know why a data analyst is always a good dancer oh boy is this a joke I I'm
terrible at these because they know how to Pivot oh no my means to remove your faiv table access we might cut that one [Laughter] youself I walk right into that one and and and by the way can you tell
us more about the road map like yeah what may be coming to jinny there's a lot I can't talk about yet uh but I would say uh the things that we've already started hinting at even at Summit are things
like for example tighter integration between dashboards and Genie where we can try to use a dashboard to sort of bootstrap a genie space so that you have to do less work in setting it up ideally
if you can ask a dashboard a question you should be able to ask the exact same question Genie and then you can ask all the questions around it so that's a sort of a big area that we are sort of
increasingly focusing on and you know some of us just making sure that as easy to use as possible um and then sort of making Genie accessible to as many people as possible like we know a lot of
folks uh don't actually want uh any random business user to be in their workspaces uh we want we know that they want business users to maybe just have access to Genie and nothing else and so
we're we're looking at various ways of making that um much more uh usable as well um and so those are I would say two of the bigger themes but then a big part of the road map that you may not see as
obvious sort of big launch features are just like refining the quality of the answers and making sure that we get the answers right by default a lot more frequently so that you as a author have
to provide fewer and fewer instructions over time and so uh hopefully that will you will just see over time as s of natural improvements in Genie ni and and and any so what's the integration between genie in the AI
assistant that we have like if you ask a business question in the AI assistant for example from notebook is it going to trigger Jenny also how how does it work oh that's interesting so you're thinking
assistant specifically yeah yeah so so we basically framed assistant as more of the companion AI thing for technical users and Genie as the thing that answers your questions for business
users and so in some ways uh they're complimentary to each other for two different crowds um I could certainly see over time things that you learn from Genie or things things that you learn
from assistant benefit in each other but we'll have to be very careful about so the governance angle making sure that the right people can only answer the right uh access to answers for the right
questions um given the level of access they have in unity calog and so again the nice thing about Unity catalog is of it will solve all of the who gets access to what kind of questions so that Genie
and assistant can sort of act as purely applications on top of that yeah then and I I think like like uh for those who are watching this recording after 30 minutes you know why chiao and his team are superstars they
are really delivering a master piece and working on one of the coolest projects ever and because like it's easy to make technical stuff but not democratizing access and makes making the data
accessible to everyone so really thank you for your hard work uh thank you I'm going to go hide behind that chair or something because no the credit really goes to the team there's a
lot of really cool really Innovative work happening across the board whether it's building the right models whether it's scaling a lot of really powerful systems whether it's figuring out the
right human computer interaction so that we can actually make the work of both the analysts and the business user as easy as possible there's a lot of fun things we're doing here and uh we're
nowhere near done yet so uh looking forward all the stuff that we're doing in the coming quarters and years all right thank you so much yeah let's give it to try you know like to day breaks go in Jenny start your first
discussion and and you know all all the magic happening for you yeah and tell us how you like it uh we always love more feedback that's how we actually grow the product faster perfect awesome thank you
CH thank you thank you