AI/BI Dashboards & Genie: Databricks AI-Powered Analytics

0:05hey today I have a special guest because as you know all my guests are very special today I have Ean with me he is the PM of aibi hi Ethan hey youf how's it going all good all good you know you I've been waiting to record this session with you for a very long time I need to

0:27thank both Miranda and shiao for suggesting your name your name more than twice and before we start can you tell us more about um maybe your role before talking about the new uh dashboard experience yeah definitely uh so yeah I'm a product manager here at Databricks um day-to-day I focus on aii dashboards uh but you know obviously

0:49that dashboards talk to Genie and you know all the other areas of the platform so I kind of dabble around in other things as well very clear very let's say clear and can you let's say walk us through how like Databricks went from the um like the old or the previous dashboarding

1:11experience to Lake View then being rebranded to aibi dashboards yeah definitely uh definitely been there's been some changes so happy to clear up any confusions and to kind of tell the whole story um but basically uh how it all started was that um data breaks used to offer a dash product produ that is now called Legacy dashboards um the product came in uh through the

1:33acquisition of redash um and you know it was meant to be kind of a pretty Bare Bones dashboarding experience built directly on the platform um and we saw a lot more uptake for that product than we were expecting you know we weren't expecting so many folks to be interested in building dashboards um but you know we heard from customers that's like they they said to us you know we love that we can visualize data directly where our data is we don't have to take it to a third party tool we love the convenience

1:56we love that we can quickly iterate um and get some quick insights share them out um but that product had a bunch of drawbacks uh the main one being the data model where uh you first had to create a set of queries um and then your dashboards would pull those queries into those canvases um and then that made sharing difficult because um to share a dashboard you also have to make sure that all the underlying queries had to be shared at the same time um and if

2:19somebody made a change to one of those queries and SE say 17 dashboards are using them like you wouldn't actually know that they're affecting like 17 dashboards Downstream so if you wanted to build this like really clean presentation ler you couldn't do that super easily um because like you know you had to manage all these different objects in different places uh so with aibi like one of the key changes we made moving into actually what we call lak view dashboards back then was we wanted

2:41to consolidate that data model so that your data sets and your dashboard visualizations and your canvas would all be encapsulated into one kind of uh Native object um and then we also took this opportunity to do a lot more you know AI augmented improvements the authorship uh so for example you can do things like uh configure visualizations of natural language we've improved the UI so a lot more things are point and

3:03click um also we move to a much more modern and fast performing visualization Library um so all of that kind of became what we call Lake View dashboards um and then what ultimately drove like the Rebrand from Lake View to aibi um is that we wanted to consolidate um Genie spaces um and L VI dashboards under kind of one uh product line so if you can think of aii as data breaks' official bi

3:27offering um and that's a new product line that contains two products so we've got aibi dashboards and aibi Genie um and I'm excited to walk through some demos today on how like those two products actually connect and and uh you can use them together yeah and for those who still don't know what's AI it's basically a new fature that data Bri release you just bring several tables

3:49and then you start like asking questions with natural like natural language questions and say hey what's my Revenue this year and you can add some instructions and then basically translate your your questions to SQL queries and it runs on the SQL Warehouse using the permissions you do have so it still enforces the same permissions and they added so many cool features there

4:12maybe we will talk about them later and last but not least this AI gen was integrated with dashboard so in case like and I I think I find this thing very very interesting because dashboards are are good they are very helpful but sometimes the dashboard may not have the answer to the question you do have and that's where for example you can switch

4:35to Genie and ask question hey what's my revenue on this specific products sold by this specific person on this specific region so it's very dumb example but this is basically kind of stuff that you can you can do and then later you can ask maybe your data analyst to build this visualization in your dashboard exactly and can you like let's say can

4:58you show tell us more about the road map and then we can switch to the RO to the ad Devo uh yeah definitely so uh as as you'll see in the demo in a second um we've done a lot of improvements uh to aibi dashboard specifically uh to make a lot of more lot more layout options possible a lot more visualizations possible um and then also we've kind of wrapped up the interactivity gaps that

5:20we had of Legacy dashboards uh so now you can do your full range of parameters like your date ranges your multi selects and things like that um and then beyond that you know we're always working on integrating Genie and dashboards closer together uh so for example right now what you'll see in a second is that Genie will be able to generate new visualizations but we love for give to give you a way to take those generated visualizations and add them directly

5:42back into the canvas so you can work with Genie as this way of creating data sets um right and also you know creating new visualizations um there's also improvements we want to make towards Maps you know so today we have Point Maps we want to make sure you can also shade things like areas of coral pleth um and then just a lot more improvements on like the fine-tuning of the canvas sizes and things like that as well but I'll talk through that kind of in more detail as I'm walking through the

6:05product perfect so you can you can start sharing showing us your demo let me do that um okay um I hope everybody can see this okay yeah uh but this is an example uh of kind of the latest and greatest of aibi dashboards uh so we're in draft

6:27mode right now um and the first thing you'll notice uh is that you know we've actually finally introduced uh multiple Pages uh so it was a common request you know previously you know if you had a lot of visualizations you had to make one very long dashboard um and that wasn't super performant and it also made it hard to navigate uh but now you can click between all of these Pages you know split them up you know I can create a new page um if I wanted to grab this

6:51Vis this visualization I can copy of my keyboard you know paste it into this new page and I can start managing things um kind of in all these different uh kind of lay options I think this one is super cool because in the past you had to create many dashboard like either stack all your all your uh your charts in the same one but you end up having a long dashboard now you can just build one

7:14with all the tabs and it's I think this is one of the most exciting features has been released in the last month I guess yeah definitely um and one thing I wanted to call out uh when I mentioned performance um is that if you have multiple Pages um and a viewer in publish mode goes to view that dashboard will actually only run the data sets required to render the page that they open uh so if you know that you know 80%

7:36of people will only ever use the summary page and you know like some keeners are going to want to look at the details um the work that's needed to render the visualizations and kind of get the data to fetch the details actually won't run until somebody explicitly clicks over here uh so you can really split this up and then optimize performance in that way um some other things I wanted to show obviously with aibi and AI being in

7:59the name um is all the different ways you can use AI to assist you in building your dashboards um now one of the things you know with building dashboards traditionally is that if you wanted a chart like this uh you would have to know kind of what to put on your xaxis what to put on your y AIS how to do coloring right and learn how to use this UI uh but now with aibi uh you can kind of just describe the chart that you want um and then we'll do our best to kind of

8:22draw that chart for you in the way that you've described it so if I wanted to recreate this chart over here and I can just describe it um I would say um I want a stacked areer chart uh showing support call volume over time uh colored by country and let's hit enter and and see where that gets us so right away you see

8:45you know that the shapes are exactly the same uh the Legends are the same I can hit accept um and the assistant has gone through and done all the configuration on the right for me based on that description um so that's one of the flows and kind of the key accelerators that you can use um with aii um as you know you build up but your dashboards some other features I just wanted to show um is that we've introduced maps in dashboards now uh so

9:08now you can have uh Point Maps uh where you can do things like color and then by size so you can show multiple data points in different dimensions um you can now uh finally uh be able to uh format your axes uh so previously you know you couldn't um so now so basically uh now you can go through I just start that over

9:30um another thing you can finally do is formatting your axes uh so now if you go into for example the x-axis you hit format um you can start choosing the types of data that you have so for example if you're doing dollar signs for currency or percentages um and you know how many decimal places you want uh you can now really get the dashboard to show up in the way that you want it to um if you haven't used our pivot

9:52tables in a while uh pivot table has also gone through a bunch of improvements um not just in the styling but you can also conditionally color cells I won't go into too much detail here you know pivot table goes pretty deep um but just know that if you haven't played with it in a while it's definitely gone a lot better and we definitely encourage you to give it another go um so that's kind of the basics of dashboards and the editor

10:15um but what we've also made key improvements to uh the sharing experience and that integration of Genie that I was talking about so if you take this dashboard and you publish it uh you'll see that this enable Genie section has been added and what it means is um based on the cont context of your dashboard so what are the visualizations uh what are the data sets um and you know what are those uh comments that you have in UC see related to all those

10:37tables that you're referencing um we will actually automatically generate a genie space for you um that is like contextualized to the data that you have in your dashboard so what that looks like is when you hit publish um and then I go into publish mode um your users will have access to this ask Genie button um and that's going to bring up this chat window um for you to ask follow-up questions so just like what Yousef was saying you know if you had a

10:59dashboard that answered you know 80% of the questions that your viewer might want to know uh but you know for example this in this case this support dashboard um and you want to know who are my top support agents you know who close the most tickets in a month um that actually isn't super easy to get at based on just the visualizations that are available so you can ask Genie to ask answer that question for you so if you wanted to say

11:21you know based on this details table uh who are my top support agents uh who closed the most tickets in January of 2023 um because I see some January timelines here um Genie is going to take that into consideration you know look at all the visualizations and the columns um and the data sets um and then essentially return me back with new

11:44answers um so my favorite thing about Genie actually is that it doesn't just give you answers it tells you exactly what it's doing um so what this is saying is that the query counts the number of close tickets for each support agent um it ranks them based on count and then selects the top three and it's telling you that it understands that January is between January 1st and January 31st of 2023 um right you've got

12:06your table here and you can resize it to get more information if it was bigger right and then um if you are a SQL fluent analyst you can go through and also verify that the underlying generated SQL is correct and we think this is a really powerful use case whereas traditionally you might have had to do things like you know call up that data engineer or the data scientists who built this dashboard to create this new visualization at new filters for you to

12:28kind of consistently get at this information uh Genie kind of shortcuts that right away and you can get answers that were you know previous not possible with just the dashboard itself so I definitely wanted to touch on this piece um and the last thing that I wanted to show as part of this demo uh is that we've uh really enhanced the ability to share dashboards specifically by allowing you to now embed with an if

12:50frame uh so when you click share uh you know you can do things that you've always could you know for example anybody in my account can view uh for dashboards that can can be accessed by everybody but you can also now take your dashboards outside uh so outside of Databricks so when you click embed um you get this really clean iframe uh that you can put into any app or you know uh tool

13:12that you have that you know supports iframes um in this case I'll show how that works um in uh Confluence uh so I'm going to go to this Confluence page um and all I really have to do uh with to get the support dashboard natively embedded into this page um is I can grab an iframe component from confluence um and it's going to ask me for a URL so if I come back here um and I know that

13:34instead of wanting the full if frame I'm just going to highlight this URL piece I can copy that come back over to Confluence paste that in um and then I can give it like a title I can give it a width I can give it a height um and as you can see uh this natively just immediately starts rendering this dashboard um and I think Confluence didn't accept my height there so I'm

13:56going to give that one more try let's do height 1,00 hit enter there we go um now we've got a dashboard natively embedded into this page I mean all the things that you know are that make a IBI really great to use are available here so your pages you know still work you know rendering is lightening fast um interactivity still happens on the client side as much as possible so these

14:18actually aren't even generating new queries it just kind of filters within your browser um and you still get things like cross filtering right and you still get all of those Snappy performances uh that that you know users typically enjoy in product so we really encourage folks you know who have dashboards that they want to broadly share to leverage embedding um the future is generally available and we're really excited to

14:40see where folks take this kind of capability and I think something important to mention here that if you want to unid your dashboard you need to make sure that the platform or the um apps where we going to be iding this dashboard requires the login with SSO you cannot just take this dashboard and I don't know put it on an open web app it won't work and this is just for

15:03security reasons to make sure that it's not accessed by someone who should not do it so it needs to be backed behind the NSO and this is this is a very important point and just matter of security yeah you said that that's a really really good point uh so with embedding today uh to view a dashboard you do have to be a datab brakes user um and then you also have to be explicitly shared this dashboard uh so not anybody

15:26who stumbles on a page with this if frame can immediately get access to your data um we have heard requests U from customers that you know it they want to manage that permissioning themselves so for example um if you want to put this into an internal portal where not everybody is on the same uh SSO provider so you can get everybody easily provisioned into datab bricks um we're creating ways to allow you to delegate authentication to that application um

15:50and then basically pass you know things that you want to filter that user by you know back to Databricks for rendering so that's something that's where we're actively working on and hopefully we can get it out for you guys soon yeah I think I think it's going to be amazing to give also more let's say flexibility to users but still now it's super important because when you use the DBC um aibi dashboards you don't

16:13spay any licensing it just the compute time and again we already handled super well the secet warehouse with the caching mechanism and like also the usage of mvs mized views and streaming tables to have super uh great performance and plus if you are maybe not as good in building charts so

16:37G like the assistant will help you build beautiful charts would would and be let's say completely not 100% but uh rely on yourself and I think this is super important and the goal of data bre as usual to simplify or to democratize access to data and AI to everyone and if you're an an analy you can just do as as

17:02like build charts with with natural language and then if you still don't have the answer you can open a i Genie and I guess you and I guess we also have like an extended road map with many many upcoming things yeah I I definitely wanted to touch on those um so so we're kind of working on we're always working on

17:24things big and small uh so some of the big items that we're working on for example are you know today with multiple Pages um when you have a filter uh that filter applies within the scope of the page itself uh but we've heard requests like hey I would love to have a global date range filter that applies for every widget across all of my pages uh so that's something that we're actively working on to get you guys Global

17:46filters uh so you can you don't have to duplicate this across every page if you wanted to have a consistent filter for that field um small things that we're working on for example um are things like being able to clone these tabs uh so today right like um if we don't have like a clone button that gets you another page so like things like that we're working on as well um that app delegated embedding is something uh that

18:08I mentioned we're working on um calculated Fields is a big piece uh so being able to you know not just take uh the fields that are available you know within your data set uh but also actually doing additional calculations like hey I just want a new column that's the sum between the column A and column C and being able to compute that within a visualization without rather than having to do that within the SQL uh so

18:31that you know if you have a goal table and your dashboard Builder isn't super familiar with doing that sort of thing via SQL um they can do that with kind of just entering basic math um using like SQL type Expressions within the UI um so plenty of exciting things happening on that front um I would definitely encourage everybody to look at our DB uh aibi release notes um we release every week uh so you'll definitely see some

18:53goodies uh from there every single time I think something import also important to mention if I'm not wrong now you can support deployment withd bundles yep that's right dab dabs came out in the past two months yeah that's that's super that that's amazing so you've seen how this uh Mass this product like evolved a lot since the uh the ga on at

19:19data a summit and that was just I think uh six months ago so I'm really I'm really Keen to see what you're going to be present at the next data and AI Summit I'm pretty sure it's going to be something amaz amazing we hope so too um actually you said one thing uh last thing I wanted to show uh for folks is um if you have um a legacy dashboard um

19:44we've also really made uh the uh experience a lot better uh for migrating them um so just real quick um I've got a legacy dashboard here right um and previously if you've tried migrating a legacy dashboard to aibi you might have seen that you know there were a couple things that didn't work uh for example your parameters might not have converted correctly so that you saw some broken charts on the other side um now uh you

20:07know when we go into you know clone to AI dashboard um we've introduced this kind of best effort conversion for your legacy parameter syntax towards the new colon style and aibi uh so I'm just going to show what what happens when I clone so for example I'm going to clone this dashboard um I'm going to open it right um and what you see is that you know not only are all your visualization loaded

20:29uh but within the data tab uh you see that you know your your data sets are being loaded properly um and if you do have a parameter uh those automatically get converted uh so in this case my date range parameters and my ZIP codes have all been previously like automatically converted uh into this colon syntax uh so when you go onto the canvas you know all those three filters at the top are now added uh yours your visualizations

20:52work um and it's just a lot more of a smooth experience um and this also is available in the API as well so um if you want to start bulk migrating any remaining Legacy dashboards that you still use and you couldn't before you know because these parameters weren't supported um just rest assure that that's now fixed that's amazing thank you thank you so much Ethan and looking forward for our to our next recording yeah thank you

21:17you