Simplify Databricks Analytics with AtScale Semantic Layer

0:00our core principle is you don't throw away data cuz it was too expensive to store we keep having more and more data you know it was very clear to me that uh a lot of companies in the future were going to move this direction every one of them can be quering data and Databricks with the outscale semantic layer democratizing access to everyone without having to do this tradeoff hi my name is Yousef and today we have a

0:22special guest it's Dave Mariani CTO of at scale hi youf thanks for having me on it's a real pleasure so please can you tell me more about first about you then about your company yeah thanks uh youf so um so I'm the CTO and co-founder of at scale uh and uh I've been in data uh big data from from

0:45the very start uh so uh back at Yahoo um I was at Yahoo when we were inventing Hadoop uh and so uh you know I worked with uh Hadoop as our sort of main data lake at Yahoo and sort of struggled to uh make that data in Hadoop available to uh the business so at at uh at Yahoo I

1:08was I ran analytics and every team had a different Tool uh we had Tableau we had micro strategy we had Excel we had custom applications we had click View and uh and ultimately you know what I saw happening was that everybody was uh was was um taking extracts so I had to create uh I had a a team of 200 people

1:31who basically took all the data into dupe and made it really small to make it work for these different bi tools for each different division each different Yahoo web property and at the end of the day nobody could agree youf on what a click was or what an impression was um so uh it was chaos and so uh as a result of that experience and just the

1:54experience working at other companies like axom um running data and Analytics um running data analytics for an ad Network it was very clear to me that there was something that was missing in the stack in the analytics stack and that was a semantic layer and not just a semantic layer because semantic layers have been around for a long time right they've been embedded typically in the

2:18bi tool um sometimes they're embedded in the database in the form of views in the database and what I really wanted was an independent layer where I could put all my business logic in one place uh and then allow any tool to access that data but get the same answers and get the same inex interface and get the same gr performance without moving data

2:43so that's what ads scale is it's a universal semantic layer it works really well with uh Databricks um and it allows the data to stay in Databricks uh without having to create extracts or Imports like typically people have to do today um when they want want to get access to data for analytics yeah I I mean I I was listening to your

3:05background it's really amazing like I mean you were there in at at yahu when when Hadoop was born and like using all those Technologies really really amazing and I'm I'm so impressed you know Yousef we saw like at the at the time you know we you know Big Data was not really a term yet this is how old I am uh but big data wasn't really a term yet but you know it was very clear to me that uh a

3:28lot of companies in the future were going to move this direction uh that they weren't going to throw away data because that was always at Yahoo that was always our our core principle is you don't throw away data um and and and people were just not collecting it back then because it was too expensive to store because all they had was Oracle they had relational databases that were very expensive and so Hada sort of broke

3:50those bounds those boundaries for us and so it was very clear to me that people would want to store more data which meant that they would want to analyze more data that that's also the right time and I believe that's that's what that's what pushes you like to work or to start up this idea of at scale yeah and so and so now we have a lot of joint customers that are getting a lot of value um and uh what I love about it is

4:13that you know there's there's no more compromises you know what we've asked business users to do is basically compromise and they're going to compromise based on their latency so they're going to get older data um or we're going to ask them to get less data so they have they have less visibility of what's happening because we're going to take data away because we can't give it to them all of them um and we're

4:36going to ask them to wait a long time to get it because we're going to have to prep it for them to get it into the form that they can actually use um and that's just like that's just wrong and so uh and so that's what we really set out at ATS scale to solve is like to say hey you know what all data to all people in the form and in the tool that they

4:58already know so don't force them to change their habits Don't force them to move to another technology to visualize let them use it if they want to use Excel which I would love to show you how uh how datab bricks appears in Excel let them use Excel think Excel there's billions of users of excel out there um and every one of them can be quering data and Databricks uh with the out

5:21scale semantic layer that that that's really amazing it's democratizing access to everyone without having to do these tradeoffs as as you you did men I'm I'm pretty curious can you show me how to connect at scale with with Databricks yeah let's do that so let me share my screen and uh let's me let me show you what the semantic layer looks and

5:44semantic model looks okay so this is what we call Design Center Yousef so Design Center is for the data Steward to basically prep the data and they're going to prep the data with a semantic model and they're only going to do that once youf so we're going to create it once and then use many so rather than forcing users to create a model um in

6:06their bi tool they're just going to be able to consume the data you see here on the right so here's our preview this is what the this is what the bi or the business analyst is going to see and they're going to see things like they're going to see like things like sales metrics organized in my case in a folder like order quantity and sales amount Etc and they're going to see multi-dimensional data I got a bunch of customer attributes about where they

6:28live and I may want to roll them up in from City and Country and state or I want to may use zip code to uh postal code to look at them um and I may want to look at my data by orders and order dates as well as shipping dates and so I have all of this available to me without having to actually understand the modeling of the data now what you see

6:50over here on the left is the actual physical data from Databricks so uh so here's the here's the main sort of source of this of this data is called it's called fact internet sales and it's just the a transaction log of retail data and you can see some of the data youf is actually key value pairs or Json

7:12data some of it you can see some of the dates are in numbers some of the dates are in strings so what we've done here in the model is we've created these calculations to be able to do things like creating a calculated tax all done virtually and we've created these Dimensions which you see in green here by creating um them as objects so the

7:37customer Dimension if I double click on it is actually another model that's made up of a geography Dimension and as well as a gender Dimension and a table coming from Databricks that's a Delta table for customers uh data and then a geography table had geography Dimension has four other tables to roll up those two different hierarchies so all this

8:01complexity is hidden from the user okay so then you asked how do I connect to uh Databricks well I already know ATS scale's model is connected to Databricks now so what I did is I just went ahead and deployed that model and now what I'm going to do is I'm going to go ahead and and switch my hats and become first just an Excel user so let's go and

8:25let's get the data in Excel so youf watch what I do here you see how I use this analysis Services connector yeah so this is really unique because with Excel wants to speak for live pivot tables it wants to use uh an analysis Services interface and so we emulate that interface which means I can just

8:48put in the at scale connection string and use my normal Windows credentials and just like that here's all my Databricks data uh and here's that model that I saw called internet sales uh and now I can create a live pivot table on it so I didn't have to model any data I didn't have to dump any data because here's that semantic model

9:11that you just saw in ATS scale see you have all the different here there's the folder for sales metrics and I can just go ahead and click on that um and now I'm running live queries and Databricks and you remember I had those different hierarchies for where my um my my customers lived where there's those two hierarchies and so now what I'm doing is like I'm getting that live data and now

9:33I'm drilling down on that data um and it's actually applying security attributes because I can only see Canada and the US uh because uh of my RO level security I've I've uh set um and then here's all my States and for people who don't want to use a pivot table you can turn these into formulas see this

9:56convert to formulas now this is not at scale functionality this is actually Excel functionality um but watch what I can do Yousef so I can go ahead now and these these individual cells now turn into or these individ that table now turns into individual cell formulas you see how it says Cube value here yeah so

10:18I can take Mississippi data and I can take Utah data and I could take California data and I can embed it anywhere in my workbook and when it comes time to refresh the data I say refresh and those cells will automatically be refreshed regardless of how they're being used or where they're being used in my in my workbook so this just like that and that

10:42fast so same thing with powerbi so powerbi um with powerbi I can go ahead and again use that analysis Services uh um um endpoint instead of actually having to understand how to connect to my cluster so I can still use that xmla interface and I'm going to do that and look Yousef I'm connecting live not

11:06import mode most people are going to connect in import mode because connecting live is not possible with unless you're connecting to analysis services or powerbi premium but look at this I can go ahead and select that model and all I have to do is Select it and it's going to connect live now so it's not going to use the direct query

11:28interface because the direct query interface is slow it's using a live connection like you see here um and just like I did there's my same semantic model here right there's my there's my metrics that you saw in Excel here's like my uh all my different uh measures that you see here um and then again here's that geography Dimension that you saw under my customers and again I can

11:52go ahead and look at my data and look at my data by uh by my geography just like that and if I look at my my model view in in uh in powerbi you can see that the model is inherited Yousef so I don't have to recreate this whole model as a powerbi user before I start to do my my

12:14analytics it's already been done for me I just connect to it and go okay so what was happening behind the scenes so let me show you like what happens uh in what I just did and on this model so I'll go ahead and select uh my query log here so here you see internet sales and here's all the queries I just ran and you can see look

12:38at this query right here uh Yousef you can see it says it it used AGS and it ran in 360 milliseconds well there's that MDX query that came from Excel and then you can see here's the query we ran against Databricks and if you notice this special table here this is an aggregate table So what at scale did is it it built off of

13:03what could have been a trillion rows uh of data in in my Databricks cluster it actually build an Aggregate and use use that aggregate to get deliver that query in 39 milliseconds so uh so and and at scale will maintain these Aggregates look at that aggregate it was six rows uh so it's so that's how we're able to deliver

13:26speed of thought queries but without without having to go to import mode or to go to extracts like with you do with Tableau or dumping data in Excel um by running queries and then copying and pasting so that's really how that's really how it how it works um youf I can you can create these models in literally

13:49minutes and then anybody can use those models and query those models in a very easy to ous Fashion using the tools that they already know and love um it's very impressive to be honest and and I'm I'm pretty sure we have a lot of folks that loves using Excel and pretty sure they can combine

14:11at scale data braks and keep using uh Excel I mean it's a m i mean we keep having more and more data but people s with the same with the same let's say serving serving part but that I mean it's really really amazing and I have another question for you Dave if someone wants to reach out to your team is there

14:33any link that I can add in the description and also if you have any uh uh page where people can learn to use ad skill that's kind of B amaz so we kind of share this with the with the community yeah a couple so a couple call outs uh Yousef so um first of all we're in partner connect so if you go to uh if you go to your console in uh in your

14:55datab bricks console you can go to partner connect and you can go for a free trial uh with that skill and partner connect so that's one way you can do it and the other way I would say is um we have the semantic layer Summit coming up on April 24th um we have some very exciting announcements to happen there um and I don't want to give anything away but look for semantic

15:17layer Summit uh just Google search that um sign up it's free it's a virtual conference um and we're going to be making some big announcements on other ways that you can try atcap um and try it for free so today go to partner connect come to semantic layer Summit and uh see all the exciting new news that we have uh for um for the

15:41product and for the semantic layer community so you will have all the links in uh the description of the video and make sure to all register and watch the semantic layer conference yeah that's great thank you thank you so much Dave thank you for showing us the value of ad scalon is very very impressive and I wish you good luck thanks youf