Delta Lake 4.0, Metrics Layer, and DAIS 2024 Announcements

0:03how you doing living the dream man yeah amazing so please tell us your feedback but what you just have seen this is history uh I love working for day BRS this place is amazing like uh I didn't know half of the things that just happened but it's just masterful Strokes uh so let's just recap the events right

0:25let's let's recap it for everybody what's happened over the past two weeks well let's go back the story goes back a little further you know when I first joined Databricks like almost six years ago now I think like five and a half years ago do you know what my first uh event was the my first like uh Community event I did was the data Mastery tour with Databricks and snowflake back then we were

0:48we were partners and I'll never forget my manager at the time told me this is the last one of these we do and I was like why uh I was so confused like what what's wrong you know snowflakes a good dat Warehouse dat of bricks does like everything else you know he's like well look at Delta if you could put a data warehouse on a data Lake why would you pay snowflake millions of dollars right that

1:10makes sense kind of makes sense so I think like it's been a long journey to get here and you could tell that snowflake has been getting pissed about us you know trying to cut them out about three years uh I joined Databricks and uh we started kind of going in on each other's Turf right yeah uh about three and a half four years ago Databricks launched our data warehouse right and

1:33people think like Databricks does everything uh you got to add like how complicated is to add a data warehouse right it's like we just heard from Duck DB you know what duck DB is you all heard of duck DB it's basically all of data warehousing all of the academic knowledge of data warehousing put into a 2 megabyte file right all of data warehousing is a solved problem we know everything about it even duck DB is a

1:56data warehouse in a box in a 2 megabyte file it's not hard Databricks did it we make an amazing AI warehouse and now uh snowflake is pissed so they've been kind of like messing with us two last year their Summit was the same week as us but in Las Vegas this year they were like you know what we're going to do they're gonna we're going to make it the week before

2:19data breaks and they if you ask me maybe they were trying to I don't know get the jump on what we were announcing so what did they do bis BS in 19 the in 90 days in 90 days not today not today but in 90 days pois would be is did they announce open source or they it just they're going to announce it see they going to open source in 90

2:42days okay so what happened just now on stage amazing that's amazing uh M mate announced that he's we're open sourcing it not in 90 days but just then on stage on the stage he clicked the button and the repo is live and the repo is live you can go there right now completely open source you don't have to wait it's amazing it's not just in just s there

3:05what's the biggest thing that we hear from customers right now like besides the catalog we solved the catalog we solved the data layer with Delta and even the war the format War is Over Databricks acquire tabular it's going to be a long road to unification but we're going to unify these these dat so we got the data set we got the governance now the governance is open but this AI it

3:28needs to understand what's going on right indeed and a lot of times the business they speak a different language right they speak a different vernacular what what do we call this right now it's all the rage uh semantic layer or metrics layer right yeah Unity catalog metrics like right can bring metrics from DBT from at any at scale amazing it's absolutely H so now Unity catalog

3:51is going to have a metric store it's going to come later this year but you you can choose uh you like do you like at scale I love them I great it's a great UI uh I think they've made great moves in making it their free edition uh now they have they Developer Edition I think Developer Edition yeah but have you ever heard of cube yeah I I interviewed them yesterday and they're

4:13part of data break inventors now if I'm not wrong dude I I had I had I I just saw that we invested in that and I was like I talk to you know as I do everything now I asked perplexity what's Cube and of course it didn't give me the right answer I meant like Cube the people that do the semantic layer and I figured out who they were and I actually went to go talk to them too I figured out it does a lot more than Cube they

4:36actually solve a lot of problems that that we have with Databricks they have a graphql layer that interprets graphql on our behalf and then commits a gives us SQL queries they also do the semantic layer what I really like about cube is that their translator translates it's one unified translator translates between MDX and SQL yeah exactly I show

4:59that yeah yeah yeah for Tableau and powerbi and Excel like it all speaks the same language right and then they have a a rest API rapper like we have an execution API but it doesn't have a wrapper around it they they have that too so whether you like dbts new metric Slayer cubes amazing support or at Scales awesome UI that is I mean this is

5:21what people were using to build cubes for the past 10 years right so like however you want to build your tube if it's the DBT way the cube way or the ATO way dat is going to support the metric storage and and Cub has also an open source version in case you want to try it really yeah I'm I I need to come up to speed on Cube I think I think I've been spending too much time with DBT and at scale I think I need to go talk to

5:43them and SQL mesh have you ever heard of squl mesh this is what I'm talking about SQL mesh is like all the rage and I no one's ever heard of it I think we need to talk more with SQL mesh and Cube just to kind of figure out what's going on but uh lots of exciting news today I mean it's still happening we walked out Delta 4.0 Delta Forno shant the I guess amazing features in there uh it's just it doesn't stop you

6:06know I was just thinking I was I was walking out before I ran into you I was like this is too much we announced way too much do you think this is too much like and I need I need I think three months to walk through all the latest features try them play with them understand how they work Deep dive on them and maybe I can be ready by September maybe maybe maybe I'm not sure we got we got to figure out how you

6:29know to wrap this up I wonder you know we uh we gave all the attendees did you know that everybody got the ability to get a test drive on Databricks yeah and that's amazing so if you wanted to try all the attendees here got to try out uh a new test drive have you ever heard have you know what what that is yeah so you have 100 bucks credit where you have all the sist

6:53offering I think have SQL warehouse and some gen features and you can get get started quickly you don't need to set up anything like the storage is there the compute is there just open it create it and you're good to go I call it super simple data brecks yeah right no more deploy hey do you know what my number one complaint was when I joined data brecks uh like

7:15all long time ago uh you can actually go back and look up my 155s deployments deployments are horrible you know that's why they hired us as field Engineers was to help customers like Stitch all this together now that dat brakes is super simple kind kind of worried about my job to be honest like what are they going to need me for if it's so super super simple right assistant gen if you can

7:38just use natural language and you could start up Databricks with just your email address and there's no Cloud configurations to mess with there's no buckets to configure and it's just there and you could just start asking for what you want semantically like it understands the semantics that's pretty that's pretty simple I'm going to have to find something else to be useful about May

8:00thanks mate