AI/BI Genie Goes GA: Deep Research, Knowledge Store & More

0:03Hey, we are at the data and AI summit at the Mosconi Center in San Francisco and I am with Richard direct director marketing in charge of the AIBI at database. That's correct. How are you doing? I'm doing good. Just tired. It's been a long day and like a even longer 6 weeks runup to this thing. It's crazy

0:27working in marketing when you put on a summit like this. It's nuts. Yeah, that's crazy. I think we have more than 20,000 people. I believe that is the case. Yeah. And even more like uh virtual as well. Yeah, that's it's just insane. And I'm pretty sure you're super pumped because of all the innovations and features we are releasing. Yep. Like this week at the data summit. So, can

0:51you tell us more about the gen part? Like what are the most exciting features we're going to be releasing? Yeah. So uh so AIB Genie actually was announced last Summit. So uh it's been out there for a while now and been in preview for a while. And the big news for Summit is that we're gaing the product. Um you know the last 12 months or so we've

1:14actually seen over 4,000 customers now like use it and use it for production use cases as well. So it's really uh battleh hardened and tested out there in the field. Uh so ready for prime time and you know so super excited to get the actual general availability of it and that comes along with um quite a few

1:36new features as well that are pretty cool. Can you spoil those features? Uh yeah, I mean I can give you a little talk a little bit about them, but um so I think uh some of the most exciting things we're doing is we're really uh revamping what we're now calling knowledge store. So this is like basically the whole um repository of

1:58instructions that uh that Genie has. Uh so we think of um the knowledge store as kind of like a local semantic layer for Genie. Um so how Genie learns typically it starts off by learning everything in Unity catalog so tables relationships comments etc. But then when you actually

2:21build your Genie space as people start asking questions you always need to like add new semantics add new examples things you couldn't possibly think of before uh you know when you when you originally created the space. So the knowledge store is a way for um uh for admins of the genie spaces to actually do that directly inside the genie space.

2:44So for example, they can add their own add and amend the their own comments on top of the comments that are in Unity catalog. They can declare joins, they can uh add new SQL examples for Genie to learn from and stuff and synonyms as well and synonyms and all of that. And then what you'll be able to actually do as well is publish that back into Unity

3:06catalog. So then that knowledge is captured centrally and can be used across in other Genie spaces and other applications and you know dashboards and so on. Yeah. In other words, you did simplify the handling of all the metadata like rather than having to do it for example in like outside G now you can do everything inside G to make sure it's much more dynamic. Yeah. And then

3:29there's a couple of really cool new features. Uh so we're calling uh them knowledge mining and knowledge extraction. So uh knowledge extraction is really how Genie learns from a business user in uh asking questions in the moment in real time. So let's say a business user said, "Show me revenue for my platinum customers." Genie said, "I

3:52don't really know what platinum customers are. I don't see it in the data." So the end user at that point can say well platinum customers are every customer or any customer that generates say above $20,000 in revenue a month. So Genie will then learn that and try and use that logic and show you the results.

4:14then the user will click yes I like this response and at that point uh all of that knowledge then gets extracted and added as instructions or or as components of instructions. So for example um the new platinum customer definition might be a new wear clause or it might be a new metric definition but that actually just at that point gets

4:37extracted from the end user just typing it in and stored in the knowledge store um for for future use. And then just as exciting is uh something called knowledge mining as well. Um uh and so knowledge mining is you might log into or the admin logs into a Genie space one morning and is basically given a hey I

5:00intercepted five queries on the same tables that this space is built on last night. So maybe a data scientist is querying those tables or some other BI reports or executing queries on those tables and it'll say do you want to learn these do you want to add these new queries to the knowledge store? So you can at that point the admin can go through and inspect them and and click yes, accept or reject. So it's just a a

5:24much more dynamic way of building uh Genie semantic model on the fly. Yeah, that's just amazing. It's like a lot of exciting announcement. Yeah. And what about uh UC metrics? Oh yeah, so we're also supporting uh UC metrics now in Genie and uh on dashboards as well. And

5:47you know that semantic layers and metric stores have been around for a while but really it's all about giving end users trusted answers. So inside any metric view, you can have very complex SQL definitions stored inside that metric view. And then you just add Genie or dashboards on top of that metric view. And then it's just a simple query of give me the revenue metric and that's

6:11all the all all that Genie needs to know about. And then behind the scenes that metric definition is translated and the query is run and the results come back as trusted answers to Genie. Yeah, that's it's very interesting because you will make sure that you have this hub where you can capture all the KPI to make sure that everyone is aligned and everyone is using the same exactly let's

6:33say like values or computation because sometimes two two teams can have different way of computing the same metric which can cause some issues because you have different values for the same thing. Yeah, I mean they're not new like you know metric uh metric views semantic layers have been around for since I started in BI which is nearly 30 years ago now. Um you know the concept

6:56was always you don't want people arguing about the numbers in a meeting. You want the numbers to match up you know um but I think one of the things that we're doing really uh at Databricks that's unique is actually uh building those semantics into uni into unity catalog. So, it's not just about the BI tools. It's about, you know, notebooks or ad hoc SQL queries or maybe it's a, you

7:18know, something like Tableau or PowerBI. They're all going to get the same numbers coming back because they're all sitting on top of a centralized uh semantic layer which is inside of Unity Catalog. And I also heard that you have uh a new release regarding a consumer role in Genie. Well, yeah, I'll tell you. There's one more thing I want to

7:41tell you about in Genie first. So, and probably the most exciting uh is this concept of uh deep research mode. So, we'll make a big splash about this um uh you know and basically what deep research mode does is it helps users ask really sophisticated questions. So in the past it's been very Genie is great

8:04at saying you know what's my revenue last month by quarter by region like asking those descriptive questions but if you see a spike in a chart or uh you want to understand like how to optimize your salesunnel you can just ask Genie like why is this number like the number is or how can I increase my salesunnel

8:26and it will actually then uh form a bunch of hypotheses and build a research plan and go off and actually test all of those hypotheses against the data and then bring everything back in a nicely presented like almost like a research paper that you can then go through and actually really you know really start to understand root cause behind all of the numbers and and even like you know help

8:49to kind of understand how you can make things better uh in the future as well. So it's much more than just like basic descriptive analytics. That's amazing. Yep. And I think I played this with this feature and I was like it just just um amazing. Yeah. No, it's it's pretty cool. I mean it's early days so we'll see we'll see what I know what

9:12customers do with it. Um but you know this is really like been the promise of analytics and and BI ever since I've been working in the space is it it not only does descriptive but it gets into prescriptive and then you know uh actually you know um you know even even beyond that. So to really help people kind of understand root cause and actually be able to act on the data versus just know what's going on. Yeah,

9:35that that makes sense. Yeah. And what about the consumer role I mentioned? Oh, consumer ro. So so yeah, so uh of course you know uh that um Databricks is a very friendly place to come for data engineers, data scientists, analysts. um when you log into the workspace um you have access to everything. Yeah.

9:59Uh but now that we have tools like AIBI which are really designed for business people, nontechnical users to use to make decisions, we have to get those genie spaces and those dashboards to those end users. Um they do not want to come to the Databricks workspace. it is very complicated and overwhelming and they'll probably log in once and then

10:22never come there again. So uh consumer uh entitlement is really a new workspace entitlement that you can give to any user or group and when they log in they'll see a very very simplified view of the workspace where they can have access to dashboards genie spaces and then pre-built Databricks apps and they can just have readon access and they can

10:44find them uh use those particular assets and get on with their get on with their job. Uh so that's that's the entitlement piece which is really just a toggle in the workspace. What we're really excited about is what we're calling datab bricks one. Data bicks one is the actual user experience itself. It looks nothing like the datab bricks workspace. It's a

11:07beautiful highly designed um uh UI that is just has all of the complexity taken away. Um, and it's just very much designed to give a user exactly what they need to answer the questions and get their job done. So we'll see things like groups of assets organized by domain. For example, so if I am in

11:30marketing, which I am, and I just want to know about campaign analytics, there'll be a domain called campaigns. So I might have 10 dashboards, a couple of Genie spaces, maybe four different Databricks apps that all help with campaign analytics. So they're kind of organized into business areas. So you might have like supply chain uh supply chain domain or um you know um sales and

11:52marketing I don't know but uh very much like tailored for business people to think in in business terms and that will simplify life of many many customers that's for sure. Yeah for sure. And you know our goal is here to get the power of Databricks in the hands of everyone in the organization not just you know data practitioners which is most of the people here you know we want to go way

12:14beyond that. So uh and the reason we call it Databricks one is because we see when it comes to data we want an organization to be one. We don't want it only to be for the privileged few. We want everybody to use data in their day-to-day jobs. So you know Databricks one is the democratization of Databricks for everyone in the organization.

12:36That's awesome. And I think you have the you know why it was called one. So this is an exclusivity. And what about uh dashboards? Yeah. So I mean um we're not making big announcements on dashboards but we're we have actually just put a a a lot of new features into dashboards as well. um you

12:59know it uh BI tools have been out there for a long timedeed and there's a lot of you know basic features that our customers expect. So um you know we've really we really feel now that we've kind of filled that gap with like you know become a absolutely uh valid BI tool for people to use enterprise BI.

13:21So, we're we're we're um with this particular um at summit, we're going to be talking about things like we've added themes, you know, multi-page uh drill down, drill across, all these different, you know, just things that people who use BI tools expect. Uh we're also introducing some predictive analytics capabilities into dashboards

13:43as well. So the ability to, you know, rightclick a trend line and get a forecast and that forecast will actually run on a real, you know, profit model that's kind of hosted and running inside the Databricks uh uh platform as well. So it's not just like lightweight models. These are like fullfeatured um like forecasting models that are running behind the scenes. U and we're also

14:06introducing something called key drivers which again when you're on a dashboard maybe you see a spike and you want to understand why it's happening you can kind of rightclick and and say help me understand this spike and um you know right inside the dashboard experience and start to see you know well you know do analysis to understand why you're getting certain trends in the data and so on. Yeah, that's still a bunch of

14:29releases like uh that's that's still huge. Like if you compare like our the uh dashboarding experience that we had maybe two or three years ago, this is Yeah. I mean, even if you compare it to uh what we what we released last summit or even last Christmas where we were, it's just staggering the amount of

14:51innovation that goes in. My job as in product marketing is to evangelize all these new features and I can't keep up with the pace of the engineering team and what they're delivering. So we actually just have to like you know put blogs together now with like 20 20 new features at a time just because the stuff is just coming out so fast. That makes sense. Thank you. Thank you so

15:14much, Richard. Yeah, no worries. It's great to see you here. I hope you're having fun. Thank Yeah. Yeah, I'm super happy and super pumped. Yeah. Right. Cool.