AtScale
Databricks and AtScale
our core principle is you don't throw away data cuz it was too expensive to store we keep having more and more data you know it was very clear to me that uh a lot of companies in the future were
going to move this direction every one of them can be quering data and data bricks with the outscale semantic layer democratizing access to to everyone without having to do this tradeoff hi my
name is Yousef and today we have a special guest it's Dave Mariani CTO of at scale hi youf thanks for having me on it's a real pleasure so please can you tell me more about first about you then about your company
yeah thanks uh youf so um so I'm the CTO and co-founder of at scale uh and uh I've been in data uh big data from from the very start uh so uh back at Yahoo um I was at Yahoo when we were inventing
Hadoop uh and so uh you know I worked with uh Hadoop as our sort of main data lake at Yahoo and sort of struggled to uh make that data in Hadoop available to uh the business so at at uh at Yahoo I
was I ran analytics and every team had a different Tool uh we had Tableau we had micro strategy we had Excel we had custom applications we had click View and uh and ultimately you know what I
saw happening was that everybody was uh was was um taking extracts so I had to create uh I had a a team of 200 people who basically took all the data into dupe and made it really small to make it
work for these different bi tools for each different division each different Yahoo web property and at the end of the day nobody could agree youf on what a click was or what an impression was um
so uh it was chaos and so uh as a result of of that experience and just the experience working at other companies like axom um running data and Analytics um running data analytics for an ad
Network it was very clear to me that there was something that was missing in the stack in the analytics stack and that was a semantic layer and not just a semantic layer because semantic layers
have been around for a long time right they've been embedded typically in the bi tool um sometimes they're embedded in the database in the form of of views in the database and what I really wanted
was an independent layer where I could put all my business logic in one place uh and then allow any tool to access that data but get the same answers and get the same inex interface and get the
same gr performance without moving data so that's what ads scale is it's a universal semantic layer it works really well with uh data bricks um and it allows the data to stay in data bricks
uh without having to create extracts or Imports like typically people have to do today um when they want want to get access to data for analytics yeah I I mean I I was listening to your
background it's really amazing like I mean you were there in at at yahu when when Hadoop was born and like using all those Technologies really really amazing and I'm I'm so impressed you know Yousef
we saw like at the at the time you know we you know Big Data was not really a term yet this is how old I am uh but big data wasn't really a term yet but you know it was very clear to me that uh a
lot of companies in the future were going to move this direction uh that they weren't going to throw away data because that was always at Yahoo that was always our our core principle is you
don't throw away data um and and and people were just not collecting it back then because it was too expensive to store because all they had was Oracle they had relational databases that were
very expensive and so Hada sort of broke those bounds those boundaries for us and so it was very clear to me that people would want to store more data which meant that they would want to analyze
more data that that's also the right time and I believe that's that's what that's what pushes you like to work or to start up this idea of at scale yeah and so and so now we have a lot of joint
customers that are getting a lot of value um and uh what I love about it is that you know there's there's no more compromises you know what we've asked business users to do is basically
compromise and they're going to compromise based on their latency so they're going to get older data um or we're going to ask them to get less data so they have they have less visibility
of what's happening because we're going to take data away because we can't give it to them all of them um and we're going to ask them to wait a long time to get it because we're going to have to
prep it for them to get it into the form that they can actually use um and that's just like that's just wrong and so uh and so that's what we really set out at ATS scale to solve is like to say hey
you know what all data to all people in the form and in the tool that they already know so don't force them to change their habits Don't force them to move to another technology to visualize
let them use it if they want to use Excel which I would love to show you how uh how datab bricks appears in Excel let them use Excel think Excel there's billions of users of excel out there um
and every one of them can be quering data and data bricks uh with the out scale semantic layer that that that's really amazing it's democratizing access to to everyone without having to do
these tradeoffs as as you you did men I'm I'm pretty curious can you show me how to connect at scale with with data bricks yeah let's do that so let me share my screen and uh let's me let me show you
what the semantic layer looks and semantic model looks okay so this is what we call Design Center Yousef so Design Center is for the data Steward to basically prep the data and they're
going to prep the data with a semantic model and they're only going to do that once youf so we're going to create it once and then use many so rather than forcing users to create a model um in
their bi tool they're just going to be able to consume the data you see here on the right so here's our preview this is what the this is what the bi or the business analyst is going to see and
they're going to see things like they're going to see like things like sales metrics organized in my case in a folder like order quantity and sales amount Etc and they're going to see
multi-dimensional data I got a bunch of customer attributes about where they live and I may want to roll them up in from City and Country and state or I want to may use zip code to uh postal
code to look at them um and I may want to look at my data by orders and order dates as well as shipping dates and so I have all of this available to me without having to actually understand the
modeling of the data now what you see over here on the left is the actual physical data from data bricks so uh so here's the here's the main sort of source of this of this data is called
it's called fact internet sales and it's just the a transaction log of retail data and you can see some of the data youf is actually key value pairs or Json data some of it you can see some of the
dates are in numbers some of the dates are in strings so what we've done here in the model is we've created these calculations to be able to do things like creating a calculated tax all done
virtually and we we've created these Dimensions which you see in green here by creating um them as objects so the customer Dimension if I double click on it is actually another model that's made
up of a geography Dimension and as well as a gender Dimension and a table coming from data bricks that's a Delta table for customers uh data and then a geography table had geography Dimension
has four other tables to roll up those two different hierarchies so all this complexity is hidden from the user okay so then you asked how do I connect to uh data bricks well I already know ATS
scale's model is connected to data Bricks now so what I did is I just went ahead and deployed that model and now what I'm going to do is I'm going to go ahead and and switch my hats and become
first just an Excel user so let's go and let's get the data in Excel so youf watch what I do here you see how I use this analysis Services connector yeah so this is really unique because
with Excel wants to speak for live pivot tables it wants to use uh an analysis Services interface and so we emulate that interface which means I can just put in the at scale connection
string and use my normal Windows credentials and just like that here's all my data bricks data uh and here's that model that I saw called internet sales uh and now I can create a live
pivot table on it so I didn't have to model any data I didn't have to dump any data because here's that semantic model that you just saw in ATS scale see you have all the different here there's the
folder for sales metrics and I can just go ahead and click on that um and now I'm running live queries and data bricks and you remember I had those different hierarchies for where my um my my
customers lived where there's those two hierarchies and so now what I'm doing is like I'm getting that live data and now I'm drilling down on that data um and it's actually applying security
attributes because I can only see Canada and the US uh because uh of of my RO level security I've I've uh set um and then here's all my States and for people who don't want to use a pivot table you
can turn these into formulas see this convert to formulas now this is not at scale functionality this is actually Excel functionality um but watch what I can do Yousef so I can go ahead now and
these these individual cells now turn into or these individ that table now turns into individual cell formulas you see how it says Cube value here yeah so I can take Mississippi data and I can
take Utah data and I could take California data and I can embed it anywhere in my workbook and when it comes time to refresh the data I say refresh and those cells will automatically be refreshed regardless of
how they're being used or where they're being used in my in my workbook so this just like that and that fast so same thing with powerbi so powerbi um with powerbi I can go ahead and again use that analysis Services uh
um um endpoint instead of actually having to understand how to connect to my cluster so I can still use that xmla interface and I'm going to do that and look Yousef I'm connecting live not
import mode most people are going to connect in import mode because connecting live is not possible with unless you're connecting to analysis services or powerbi premium but look at
this I can go ahead and select that model and all I have to do is Select it and it's going to connect live now so it's not going to use the direct query interface because the direct query
interface is slow it's using a live connection like you see here um and just like I did there's my same semantic model here right there's my there's my metrics that you saw in Excel here's
like my uh all my different uh measures that you see here um and then again here's that geography Dimension that you saw under my customers and again I can go ahead and look at my data and look at
my data by uh by my geography just like that and if I look at my my model view in in uh in powerbi you can see that the model is inherited Yousef so I don't have to recreate this whole model as a
powerbi user before I start to do my my analytics it's already been done for me I just connect to it and go okay so what was happening behind the scenes so let me show you like what
happens uh in what I just did and on this model so I'll go ahead and select uh my query log here so here you see internet sales and here's all the queries I just ran and you can see look
at this query right here uh Yousef you can see it says it it used AGS and it ran in 360 milliseconds well there's that MDX query that came from Excel and then you can see here's the query we ran against data
bricks and if you notice this special table here this is an aggregate table So what at scale did is it it built off of what could have been a trillion rows uh of data in in my data bricks cluster it
actually build an Aggregate and use use that aggregate to get deliver that query in 39 milliseconds so uh so and and at scale will maintain these Aggregates look at that aggregate it was six rows uh so
it's so that's how we're able to deliver speed of thought queries but without without having to go to import mode or to go to extracts like with you do with Tableau or dumping data in Excel um by
running queries and then copying and pasting so that's really how that's really how it how it works um youf I can you can create these models in literally minutes and then anybody can use those
models and query those models in a very easy to ous Fashion using the tools that they already know and love um it's very impressive to be honest and and I'm I'm pretty sure we have a lot of folks that loves using
Excel and pretty sure they can combine at scale data braks and keep using uh Excel I mean it's a m i mean we keep having more and more data but people s with the same with the same let's say
serving serving part but that I mean it's really really amazing and I have another question for you Dave if someone wants to reach out to your team is there any link that I can add in the
description and also if you have any uh uh page where people can learn to use ad skill that's kind of B amaz so we kind of share this with the with the community yeah a couple so a couple call
outs uh Yousef so um first of all we're in partner connect so if you go to uh if you go to your console in uh in your datab bricks console you can go to partner connect and you can go for a
free trial uh with that skill and partner connect so that's one way you can do it and the other way I would say is um we have the semantic layer Summit coming up on April 24th um we have some
very exciting announcements to happen there um and I don't want to give anything away but look for semantic layer Summit uh just Google search that um sign up it's free it's a virtual
conference um and we're going to be making some big announcements on other ways that you can try atcap um and try it for free so today go to partner connect come to semantic layer Summit and uh see all the exciting new
news that we have uh for um for the product and for the semantic layer community so you will have all the links in uh the description of the video and make sure to all register and watch the
semantic layer conference yeah that's great thank you thank you so much Dave thank you for showing us the value of ad scalon is very very impressive and I wish you good luck thanks youf