Lakeflow View Dashboards: Self-Contained Models & Secure Sharing

0:01hi I have two special guests with me today ladies Spurs hi thank you for having me Yousef my name is Miranda I'm a product manager here at Databricks who works on a few different things Databricks SQL um one of which is L VI so very excited to be here kind of chatting about uh Lake View dashboards and kind of some of the motivations for creating those as well

0:23as hopefully some of the fun stuff coming down the line amazing what about you hi thanks for having me youf so I'm matz I'm a solution architect bed of France and the leads me for data breaks SQL uh for Europe and um so couple of experiences on

0:45how SQL is is leading the uh the charge on on The Lakehouse point of view and a couple of feedbacks on concrete experience that we have uh um from customers amazing so I think the first question maybe for Miranda why did we uh do this switch

1:07from the to Lake View yeah definitely well before I kind of get into the Genesis of Lake View maybe it's worth rewinding a little bit and even sharing a little bit more about kind of the initial motivations for for DB SQL itself um so you know playing the way back machine one of the things we noticed you know data break started as uh you know manag spark for Engineers

1:30building pipelines then kind of branched into having solutions for data scientists looking to kind of conduct some of their analysis and forecasting work as well as ml Engineers um so you know the original intent of Databricks and manag Spark was not first and foremost to be you know any sort of bi serving layer right um but what we started to notice a few years ago was

1:52that more and more customers were doing a ton of tuning on their spark and changing configurations over and over and over again um so they could use um you know those interactive clusters with a tableau or a power VI um and we were getting tons of feature requests for how to make that easier um and hey you know could you publish recommended sets of configurations with each release and like think about this this kind of

2:15use case as a little bit more front and center um so what we kind of looked at was like hey there's probably an opportunity here kind of given the tuning you did for bi and SQL workloads uh to offer kind of a much more managed experience on top of kind of interactive clusters so that's why we launched uh SQL warehouses or SQL endpoints originally um those were really targeted

2:37at providing kind of a turnkey experience for using Databricks as your data warehouse with bi tools um so certainly that was kind of our our initial intent with SQL endpoints um and actually very early on we were not intending to kind of have like a different you know user experience for that that type of compute right um but you know we had just acquired a company called called redash at the time

3:00um and they had this great experience for kind of SQL analysts a SQL editor lightweight dashboarding tool some alerting um and we said you know hey nobody should have to set up like a whole connection create a pat token just to get a a pie chart of a Boolean field to make sure everything looks right so we said okay well let's just integrate that um give folks options kind of in the first party experience before they

3:22get uh get into their more production workloads um but what we were surprised by in doing that is well one not just happy to see that data B SQL as the kind of the compute layer took off um but we were also starting to get tons of feature requests for that first party experience right so for the dashboards that we kind of provide for just provided out of the box for uh the editing experience the SQL editing

3:44experience we wanted to address those so um what we actually ended up doing when we were kind of looking through a lot of these enhancements you probably saw early on we were shipping shipping improvements uh very regularly to those dashboards um but at a certain point we kind of got to a place where we' need to change the data model to satisfy some of these requests and if we're

4:07going to change the data the content model you know that's something that we as Databricks are not just going to surprise do overnight to anybody right we want to make that a very um very upfront change something that people are really aware of and hopefully something they're excited about um but not something that's going to be like a a surprise um so when we were kind of thinking about hey what's the next evolution of dashboards you know we're hearing that there are all these use

4:29cases for you know using the first-party experience in conjunction with you know a third party tool that might be kind of like the system of record and production whether that's looker thought spot powerbi Tableau um at many of our customers it's actually a combination of those um but you know there is a benefit to having something that sits right side by side with the data for some quick validation or doing something really

4:52lightweight and then sharing it out um and we want to make sure we were continuing to kind of meet the needs of folks with those use cases um so with Lake View and as we were thinking about kind of the next step for our dashboarding experience internal to the platform um there were a few things we wanted to make sure we are addressing right so first and foremost um and I hope a lot of the this

5:14resonates with a lot of the folks on the listening to this but uh we wanted to make sure we kind of moved away from that uh dashboard as a reference to everything that lives on the query model um that was challenging for a couple of reasons one sharing right you want to share the dashboard and all the series but you have to kind of round robin around 15 objects instead of just hitting one share button um same thing with moving it uh the other thing we

5:37were hearing was you know an ask for uh you know additional features where it really kind of helps to have a self-contained definition instead of a potential uh circular reference things like hey I want to check my dashboard into into into get via repost you know if we have kind of references on references on references that becomes a lot more challenging to do in a way that

5:59would give that kind of clean Enterprise ready experience our customers expect so that was kind of the first motivation on let's make sure Lake View each dashboard is self-contained it has its own data model it can be treated as an independent uh element and you don't have to have all this nest of references and of course I mean you guys you guys have started to use Lake View quite a bit one of the things we really

6:21encourage there is instead of saying like in the old world hey I have this one query that's like super trusted and and Yousef wrote and he knows everything and now Matthew and Miranda should go off and use it in all their dashboards well no like Yousef should just make that a materialized view in unity catalog and then Miranda and Matthew can go off and use it at all their dashboards um but you get the extra richness of things like lineage table

6:44comments uh column descriptions and and those are all managed centrally and governed in a central way not kind of in a different way because it's a query object versus a data object so that was kind of the first motivation uh also want to do a big refresh of our visualization so very excited to see that there um and not just a refresh of them to make them a little bit more modern and performant we've kind of made

7:06a few a few really important updates in that area but also to you know start to integrate uh llms so today if as long as you have uh llm kind of capabilities for data breaks enabled in your account you can describe the chart you want in English give me a bar chart of monthly sales and then based on your data set you know we'll just autogenerate the bar chart you want to see you can accept

7:28reject or if it's close enough you can just make one little modification through the manual configuration um and you're off and running and of course we plan to layer on more and more over time um and then the last place we really wanted to make sure we're kind of focusing on improving is actually you know being part of the platform right so there's a lot of great benefits that come from being right inside Databricks whether that's lineage whether that's

7:50getting us to a place where we can kind of say hey Auto speed up my stuff by making a materialized view under the hood and don't make me think about it um we want to lean into all those opportunities um and really highlight those as strengths and and accelerance for for folks building dashboards yeah I mean this this integration with UC is amazing I've seen how you can leverage this lineage to see

8:12for example if this table is called on a specific dashboard which one uh if it's this table is called on a specific query uh the governance over the tables but also the dashboard I mean it's a super super great experience and also how easy it is now to build dashboards because before needed to write queries to store them create charts and then bring this

8:36chart on the dashboard on the dashboard while now everything is within the dashboard and and one of the things we're hoping this helps with you is exactly what you said which is you know as you're sharing these dashboards around the organization that kind of data Providence and the story of like hey I can trust this look this that this table was updated this morning the data is current is super important right um

8:58and that's another exciting thing I forgot to mention about Lake View dashboards is we really tried to lean into making sure there is kind of secure sharing across the organization without having to require everyone to be in a workspace right we were hearing from more and more Enterprise customers like hey this is awesome I'm doing kind of a workspace per team I don't really want to give everybody access to every workspace it's not really necessary uh just to consume a dashboard um or maybe

9:20I have some folks that I really just don't want to see you know certain complexities that might exist that are for data Engineers if they're just consuming dashboards um so that's something we've now been able to achieve with Lake View is those dashboards are sharable throughout the organization um as long as the you users are registered with the data Brooks account it doesn't mean they have to have a workspace they don't have to have data access computer access right um but

9:43it does allow you to kind of say like hey Matthew's in my my organization he's not in this workspace but guess what he can still see this dashboard and only Matthew so if he sent you the link and I hadn't shared it to you you know you're not going to just automatically be able to see all the data so we're really excited about kind of getting that that sharing experience going and exactly what you said that's kind of also a real benefit of having it integrated with the platform and having that data Providence of like oh I can see that Miranda

10:06created this dashboard and Miranda owns these three tables that were used in it and this is when they were last updated and here are the tags that say they're certified and you know off and running one of the interesting things I found with Lev view is the fundamental difference in terms of user experience that's been approached with approached in terms of no code um I'm actually interested to have a little bit more information on how you came above and what's your philosophy around the

10:28approach to no code low code specifically around visualization around data braks sure so so there's probably like two pieces to well three pieces to speak to two for the author and one for the consumer um so for the author there are kind of a couple there are two areas we want to push on right like one is data set authoring the other one is visualization authoring um and and we

10:51started with visualization Au authoring to kind of refine some of the approach and the mechanics and and kind of make sure we're getting feedback on exactly what helps with that uh that experience as well as kind of hearing how users are are thinking about it um because it's interesting right like the most common case we've heard and certainly something we want to evolve to support is like great I did my data set you know select star from this materialized view um and

11:14now I want to be off and writing building my viz well but I usually want to add a filter like oh this chart is just for this product line or that chart is just for that product line right so one of the things we're starting to think about is how do we let you kind of get your starting uh starting data set and then maybe apply some of those things that would be data filters but at the visualization level um instead of having to go back and modify the data set so you know that's one like

11:37explicitly at least so that's one of the things we kind of wanted to start with the visualization level to get that sort of feedback because if we had started with data set of like hey here's a ton of different ways to point and click and and you know visually Define your data set but then it actually turns out that like what users are asking more for is being able to kind of do those modifications viz first from the canvas thinking about it that way you know we

12:00might have ended up in a very different place um so certainly we're going to continue to push on both of those areas um but that was kind of an interesting nugget that came out of um the direction we started with and then of course we want to get into the consumer experience being a lot more interactive and you know certainly yes parameters filters they're coming we want to do cross filtering we want to do drill downs we want to do all the things everybody's

12:22used to in a dashboarding experience right um but we also want to empower and this is you know alli's uh Al's favorite use case so certainly we care a lot about it but hey great you sent me this uh dashboard that's broken out in fiscal quarters I want to change it to see a monthly view how do I just type that in and say you know change all these from what the fiscal fiscal quarters to months we'd love for every consumer to

12:44be able to do that and to kind of reshape the data regardless of whether that was like an aggregation applied on the visualization or something actually in the SQL itself um so that's something we're definitely very much thinking about and then the other piece we're thinking about is you know Das boards are great for kind of looking at what happened what's happening now um as well as kind of like to that that case I just

13:06gave you moving within kind of some of the constraints that are already set up there like you know change this color change this filter you know modify in these ways but we also have this great new capability called Data rooms that's really focused on kind of enabling the business user to interrogate their data in ways that go beyond what was kind of preset up in the dashboard so certainly I think one of the things we're really excited about pushing on you know in the

13:28coming months is establishing kind of a really clean um kind of connection between those two two experiences so that you can go like hey great I got a slack of this dashboard and I saw this interesting thing okay I'm going to go look at it now I want to ask a new question um and making sure that whole Loop is as seamless as possible um including potentially like oh I just found this new interesting Insight that

13:50probably should be on the dashboard I'm getting slacked every day let me go ahead and update that and get it back in there and you know that link oh go ahead I was just going to give an example what I use for the first time data rooms I remember I downloaded um uh I think a data set of uh the Chargers across uh the Paris region and I mean not Paris region I

14:12think it's it's within the country within France and I I I wrote a query like hey uh show me the aggregation of chargers by Department in and and I specified the department should be the Paris department and somehow it figured out that the department should be 92 93 75 so it it get all this intelligence

14:37and then it plug it inside and I was able to have this visualization very in a very easy uh way which means anyone from I don't know it can be a sea level can build dashboard just by asking question and this is this is super super amazing awesome I'm glad it work and that plot that plot feature with the with text although we get a

15:01few errors from time to time that that's a that's a very big differentiator I think for people who don't necess know how to use and visualizations very well um because it gives you easy options and I found that a that's a game changer in the way that we approach uh data analysis on data braks um and that I know coming from a lot of customers

15:23that's that's definely been a big uh a big change uh the whole code low code approach where I just give it a data set and then I can drag and drop and start playing around with it uh I want to do something but I'm I'm not likely sure how to do that but let's try to see I I can try that prompt uh I know it's still a little early game but I think that's actually kind of very cool and

15:46definitely appreciate by customers who uh who who really like it because of its easy use well got to say previous approach where we had to do the queries had it was more of a specialist approach uh whereas yeah this is I think definitely something we can finally put into the hands of business and users who don't necessarily need to know the technicalities underneath underneath the

16:08hood and I'm also curious to know uh if we're gonna have in the future uh a tool to migrate a dashboard from a workspace to another one already there Yousef like we're doing this we're doing this quick for you um yes so we have two two things that we're really excited about that just came out one is a migration helper

16:32to go from data workor SQL dashboards to Lake use that's the first thing we're really excited about and then the second one is that export import between workspaces so whether you want to do it through the UI you know for example maybe one of you guys made an example you want to send to one of your customers you can just export it they can go ahead and import it right into their own workspace um or you could imagine you know I want to do this programmatically like programmatically I

16:54want to export all of my dashboards from this test workspace and and I want to you know back the back up those definitions somewhere so that if you know somebody deletes it they want it later I have it or maybe I want to move it to a higher environment um and and kind of get those dashboard definitions into kind of my own G provider and then have my cicd pick it up those are those are now available to everybody in Lake View which we're really excited about um

17:17and certainly like when we talk about kind of that that g experience we are looking at a repos integration in the future 100% um but today you know certainly we have some folks who are already starting to kind of uh get things a little bit more productional IED and checked into their normal processes leveraging that that export import capability that's really amazing I don't know how I missed this uh this this info

17:40I mean I used you just got to invite me on this show I'll give it to you live it'll be great I'll come every week I I mean I'm I'm really impressed by how all the ecosystem is working together because they started with this with the SQL where house engine they added the materialized fuse uh streaming

18:03tables now with analyst they can use streaming tables materialized fuse all this experience and then they can use those objects to build dashboards uh there this orchestration now you can also orchestrate um uh SQL files with uh with workflows I mean it's really amazing and plus the others I don't know I think

18:26it's also one of the coolest features like and I always push my customers to use this others because let's see I don't know you're training a model and you have the quality the quality of your models you can just have an alert whenever the I don't know the score is below uh 80% you receive an alert which means it's offload you of this time to check the dashboard every day just go

18:49and check it whenever there is something wrong yep exactly that yeah everyone the right tools they need for for kind of making their lives as easy as possible and we want to hear how we can keep improving those I think out of that another interesting point to highlight is also in the improvements we've done on the backend side specifically on this datab BR SQL serverless uh which has made the

19:13use and in terms of analysis and the power behind uh our dashboarding capabilities are much much much better um we've made tremendous leaps in terms of both improvements in terms of latency to start up the our warehouses in terms of latency in terms for for short for for short queries in terms of throughput uh we now have a tool that is

19:37definitely first class to be able to handle all kinds of different use cases on on the analysis side um and I think that's a big game changer compared to where we were at about a year ago um there's nothing stopping customers now from doing all that work directly on on Databricks we have a first class uh

19:59compute SQL computer engine on top of which uh we're building some great visualization capabilities and I have one more question for you Miranda um yeah have we discussed about all the features that are com coming in to the road map or is there any hidden features

20:22maybe uh well we're we're really excited about a number of different different elements coming out uh imminently so what we're recoring this February 12th you know in the next few weeks I'd expect to see a few uh really exciting updates land um so for example you know on the visualization side a few things that are just going through kind of final stages of release or a couple new

20:44visualization types so combination chart being able to have bars and lines together that's a very commonly requested one um histogram heat map uh so those are are kind of working their way through the final stages of release um as are uh the support for scheduling like you dashboards and optionally adding subscriptions so like you know periodically send a PDF snapshot of this

21:06dashboard to this receiver list um or you know without the receiver the email receiver list just periodically schedule this dashboard to refresh to warm the cach so that you know as all the users are kind of opening it up they have as Snappy as a load experience as possible so those are kind of working their way through right now um you might have noticed a few really exciting um like updates to the publish experience

21:29recently that was kind of something we heard was a big pain point before which was like Hey great you know you use have made this awesome dashboard Matthew and I and hundred other people are looking at it every day oh you need to make a change maybe you don't have time to finish all those changes in one little one editing session and you don't want to leave things looking broken because the hundred of us that look at it every day are going to not find that

21:51as helpful um so now you can just modify away and draft and you don't have to do any you don't have to kind of interrupt The View that that Matthew or myself or any of the other folks in the organization are relying on every day um and you don't have to make all your changes in a hurry right take your time work on your draft have others review it and then when you're ready you can republish um so there were some updates that recently shipped to that experience to kind of improve um but I'm happy to

22:15kind of chat on a couple of the the investment areas that we have in the kind of coming quarters I will say you know the one of the top of Mind things for all of us is making sure we get to G um and everybody kind of knows that that stamp is there and closing out some of the capabilities we need to make sure there um so certainly you know I mentioned kind of scheduling coming out including the ability to kind of warm the cach and again help with that

22:37initial load experience we're going to do a few more things there as well to kind of be a little smarter about kind of reusing results that are reasonably fresh and available to you um you know we made some initial steps with API support for export import um or sorry export import and Publishing uh we certainly want to continue on that so we'll continue to work uh integrate with additional workspace apis for

23:00permissions crud bringing apis online to kind of manage schedules and subscriptions uh unpublishing all that sort of fun stuff um you will also start to see kind of Lake you become more of the default experience throughout the platform so whether that's you know I'm in the catalog Explorer and it's like hey this is an interesting table let me like jump off from this table and create a dashboard that'll be there um and the

23:22the all the kind of create and AD menus you'll start to see like View kind of being the primary uh option um and certainly kind of having an uh an opt opportunity to import existing work from the SQL editor or from notebooks right like we know that uh it's not day one for every data breaks customer they have a lot of great work that they've you know invested in in making over the years and we want to make it easy to reuse that um with the new stuff uh so

23:45certainly we're going to kind of look at those paths um and then you know rounding out some of the other capabilities that we all knew in love uh we all know in love across the platform whether that's you know audit logs um parameters get making sure that as we kind of we have export Imports we serialize the definition but let's get that integrated into repos let's make sure there are terraform scripts for it

24:07um so definitely closing all that out and continuing to add to the visualizations we know we need you know whether that's Maps or box plots Etc but kind of continuing that catchup that'll be another theme um but the exciting things we're we're kind of thinking of is like net new Lake View only and you know you're not going to see these in the next couple weeks but work is underway kind of scoping and designing this um certainly embedding is top of

24:29mind you know I think that's kind of one of the most frequent requests we've gotten from customers is great I can kind of share the Standalone dashboard but actually I want to like embed this dashboard in my own web app or website how do I do that right um that's something we're starting to push on um certainly also thinking about multi-page again it's great to have like a single page dashboard but at a certain

24:51point you know you maybe don't want 50 um visualizations all on one giant long page kind of helpful be able to kind of organize that into different groups and different pages so that's very much on our radar to kind of start thinking about and then you know a lot of the other things we've uh you know we've talked about as you know just kind of something we want to make sure default experience is available to everybody

25:13whether those are you know cross filtering like let me click on my pie chart and then have that filter apply everywhere you know let me have drill down uh how do I kind of make sure this is optimized so I can put it on like a TV or a monitor in my office and not have it log me out of Databricks every 10 minutes right um so certainly thinking about those but the other area you know so kind of three buckets right

25:35like you know the ga Readiness bucket the kind of Plus+ let's make sure we have kind of a you know a good a good experience in these like expected areas um given an experience with other tools and then the last one is certainly continuing to push on some of those um AI elements and and kind of leveraging llms a little bit more like we talked about earlier uh you know the viz configuration via natural language is a

25:58great start but we're excited to enhance not just that experience but kind of like remove some of the barriers to what you have to think about where how as an author um and then also you know thinking about the consumer experience whether that is directly in the dashboard or also kind of getting that path online to data rooms um so a lot of exciting stuff down the pipe but those those things I mentioned in the next few weeks should hopefully be be

26:21exciting uh exciting stuff and then you'll have to have me back on here and I can I can do the next two weeks publishing publishing I think is going to be a big one here that's that's a big ass that we're getting be able to share easily a lot the work that's been done on data breaks and bring it out to um production applications where a lot of the work has already been done and then they want to be able to push it out in

26:44and expose it directly to the end use I think that's going to be a big big big and definitely a big half that we're seeing from customers yeah I believe that's all on my side any other any additional questions Matthew uh now so far I think we've gone over a

27:06lot of different uh things and uh uh maybe I think maybe next time we can uh spend a little bit of time doing a bit of a demo to show how this can be easily used that'd be awesome Yes um other thing I would just encourage everyone listening uh we do have ideas at datab bricks so ideas. datab bricks.com great way to kind of give feedback to the

27:28product team not just about Lake View but about all sorts of uh different experiences and capabilities in the platform we do read all of those and we do often reach out and kind of follow up and try to understand a little bit more about what you're trying to do so please continue to leverage that if you are or start leveraging it if you haven't it's a it's a great resource thank you so much Miranda thank you

27:50Matthew and don't forget to subscribe to the YouTube channel it's mandatory bye