Delta Lake 4.0: Connect, Clustering, Variant Type, Type Widening

0:04Hello everybody and welcome back to another episode of High Performance Delta. I'm joined again with Yousef uh who's a Databricks like Savant and I'm just super happy to have you here. Um and then I'm Scott Haynes. You've seen me on the show before. The only thing that's changed is I'm no longer at Nike. I'm now a developer relations engineer for a company called Buff. And with that said, uh, today you and I

0:28are going to go over like our favorite new features of Delta Lake 4.0. And, uh, with that, we're just going to start to kick off the slide deck and walk through things. So, I hope everyone is excited and, uh, happy Tuesday. So, what's new in Delta 4? There's a lot of new things in Delta 4. There's also a lot of new like new and improved things that were already in Delta 3. And so as we walk through all this, I think like one of

0:51the things to kind of keep in mind is that um everything keeps on getting better with every release. Performance keeps on getting better in every single release. Um and so things that you know felt, you know, might feel old because you've heard the name before are getting better as well. So during today's session, we're going to also, you know, recap a little bit of that as we go back through liquid clustering again. But Yousef, is there anything you want to say before we kick off?

1:13>> No, I'm just happy to be here for the second time. >> Awesome. Cool. All right. Cheers. So, I guess we'll uh go directly into the slides now. Uh starting off with Delta Connect. People have talked about Delta Connect before kind of in passing because this has been essentially like in preview for a while. Delta Connect is essentially just a plugin on top of Spark Connect. And Spark Connect came

1:36out a while back in the 3.4 for release of Spark which allowed you to have a gRPC connection from essentially like a local thin client that felt like a normal kind of like local driver in Spark up to a remote server which didn't feel much different than a typical kind of you know spark master or anything else like that. So technically like in theory a lot of what is being done feels

1:58the same. The only difference is a protocol is just all protobuff back and forth. The neat thing about this is that rather than being like essentially stuck in the JVM, you can now create a Rust client, you can create a Go client, you could, you know, whatever comes out next, create a Zigg client, doesn't really matter as long as you have the capability of being able to handle the protocol and then decide to do things within like the realm of what is

2:20available inside of Delta. If you want to uh hit the QR code, you can. Um, like if uh we're going to share this out anyway, so you can take your time as well. if you want to go and take uh you know take a look into like the docs as well. The QR code essentially is just a pointer to the Delta Connect docs. So the cool thing it decouples your Spark architecture which is nice. It makes it easier for you to actually upgrade from

2:43versions of Spark because your thin client doesn't need to know a lot about how Spark works. It needs to know a lot about how data frames work. And so all the magic really happens once essentially like an unresolved logical plan makes it up to the server. Like I don't have a cool pointer so I can just move my hands back and forth. But if we take a look at like the overview like architecture on the side, we essentially have like our Spark session which is like within like our Spark

3:06client. That's our thin client. It feels like a driver program. Let's just call it like it's like oh yeah that's our driver program but actually the real driver program is still occurring up at the server. So essentially we sort of have two drivers. One's a remote controller for like the server side driver. So your client will create a data frame and it'll just pass messages back and forth up to the Spark Connect server. Um, and then from there, if you

3:28have a normalsized Spark cluster, all the distributed computation, everything else will continue to work as expected. The only difference is now with Delta Connect and the new plug-in, you have the ability to actually communicate to your Delta tables, do transactions, do merge operations. It's all neat. Um, and so we're going to go into that in a little bit as we go through the actual demos. So essentially dropping down a little bit. We talked about kind of like

3:50the sort of like the you know the bottom to top architecture. Now we're just looking at left to right. It's pretty much the same exact thing. But if you think about like the thin clients, right? It can be created in a language that doesn't currently support Spark natively. Uh the other neat thing too is that it's fairly easy to drop all of this into like a Jupyter type notebook. So if you want people to be able to test things out quickly, that's awesome. or if you wanted to create like a you know

4:13lang language specification or anything else like that to you know create an you know a plug-in for IntelliJ for example there's neat things that could happen within the ecosystem because the client itself is tiny all it needs to know is how to create a pertabbuff message then send it to an endpoint that knows how to receive that pertabbuff message which is kind of neat so with that said we're going to move on into a little bit

4:36of kind of like a preview before we get into the demo so we talked about how we have a client We talked about how we have a server. For us to be able to connect the two of them, we actually have to run two processes. So for the connect server, we essentially just need to be able to go and run that process. So I hope the command is showing up big enough for people to see it. If it's not, we're going to share the slides out. Anyways,

4:59um I'm also sharing the actual source code for all of this so anyone can download it and take a look at it as well. But if we take a look at running the server, um I don't know if I have a highlighter, but where we see like command spark spin spark connect server um.sh, all we're doing is essentially adding some packages and one of the important ones is the delta connect server package. So that's an extension to our session that allows us to be able

5:22to run the connect server, which is essentially like an extension of spark connect itself. Aside from that, we also need like command and relation plugins to be able to understand how to essentially trans like transform a command that Spark Connect understands to something that the Delta plugin understands. Um, and it's much easier

5:44just to see it all working. So, moving on to the client. The client's essentially the same thing that we saw for the server. It's like, yeah, this is super exciting, right? So, I've highlighted here the actual like Delta like connect client. So, as we start our local, well, this what I was talking about as like our local kind of Spark session, we're just basically spinning up a normal Spark session with a thin client. So, our Spark session actually is really just a conduit for sending

6:08messages back and forth to the actual server side Delta service. And so, with that said, there's really two things. I said there's like one tiny modification. There's really two. So, everything has to run. So, it has to be running. The server has to be running. So it's already two things, but the tiny modification is that we have to identify what the Spark remote endpoint is. So if

6:30you've ever connected to like, you know, spark slash, this is really no different. It's just uh sc for spark connect slash with like the host name and the port number that you're going to connect to. The default is 15002. So that's been left off in everything within the demo other than when we actually like just, you know, talk about things in the readme. Um, so all you

6:53have to do is add an environment variable. If you're like the environment variable, I don't want that because I want people to know what's going on. Then on the Spark builder over on the right, you'll see like builder. And you just pass the remote endpoint and then your session ends up being like a gRPC session rather than one that's just speaking like the typical Spark protocol. Um, and then I highlight that as well. Like if I do type Spark session, we'll see that it's SQL connect

7:16session Spark session. And so there's just a little bit of a difference, but everything else feels the same. We're now running with gRPC instead. All right. So how does this all kind of come together? We're going to look at this in the actual session in just a sec like when we run the live demo. But if I take a look at like a normal kind of dataf frame type operation. So say I have a source paret table and I want to

7:39convert that to a delta table and then I want to save that on a path. Like it's a typical simple operation. So I've called it out. The proof's in the pudding, right? It's like we're going to go test this. We're going to make sure that it works because like Delta Connect is uh still in the its early days and it's considered experimental. But while it's experimental, it doesn't mean it doesn't work. It works really well. But the interesting thing is if you've taken a look at like any kind of like logical

8:02plans or like kind of like a dump of operations before on like the Spark side, if you've been debugging, it's neat to be able to take a look at kind of the subtle differences in like the planning. So on the right, we'll see like Spark Connect. We'll see we have a session identifier because I might have a session, Yousef might have the session, you might have different users with different privileges. And so each session has own capabilities and permissions as well. And so I'm doing

8:26something on my behalf, you might be doing something on your behalf. And the same kind of concerns of like any sort of shared clusters come into play like what's my Spark user have in terms of capabilities. So anyways, without further ado, we're going to start to transition now into the actual demo and we're going to hope that it actually works. Um, but we're going to give you a little bit of time if you want to uh take a look at the QR code that's going

8:48to be a reference to the Delta Connect playground, which will allow you to go and actually test everything that we have here on your own as well. There's some readmes. I'll walk you through what that actually means, but we're going to get started with that right now. >> May God of demos be with you. >> Thank you, sir. Um, all right. So, starting off finding my finding the mouse. So, one of the things that I wanted to just kind of uh call out real

9:10quick. So, there's a lot of data and that data has to be downloaded externally to be able to run everything that's uh currently in these tests or in this like demo environment. Um, and we're doing that because we don't want to blow the download size for people and then people can opt in to either decide that they want to run with the e-commerce data set or they can run with anything else that they want to run with as well. Um, at the end of the day,

9:32we're just taking these CSV files and we are going to be reading them into So, we're going to read our CSV files. We're going to transform them into paret. We're going to write the parquet first to show you that Spark Connect is working as expected. On the second pass, we're just going to take the PAR, we're going to write it to Delta to show you that Delta Connect is working. And then we're going to conclude by showing you what actually happens when um we read

9:54everything else back um with a couple transformations. So without further ado, we're going to go and run the Delta Connect server. Uh you can tell that the Delta Connect server is running when you get to the Spark Connect server started at blah blah blah blah blah 15002. On the other side, we're going to run the Delta Connect playground. And this is just a Jupyter Lab

10:16environment. And that's going to get up and running. And then I need to go and find this over here. So we get to the local lab address versus like the internal Docker lab address. We're going to kick that open. So right now we're starting off with absolutely nothing in our data set other than like our raw CSV. So we're going to walk through everything uh just end to end. So I'm going to first go through I'm going to create my new session. Um and then from

10:39there we're going to take a look at the type session. Make sure that everything is exactly the same as we expected. So PISpark SQL connect. That's awesome. We're go bring in our types. This part's really fun. Um just walking through it. uh our data set dur is basically set to our spark working directory inside of the docker container which is op spark workd and then data sets is bringing an

11:01e e-commerce raw. So all we've done is basically done like a virtual mount of the data set ecom um when we ran the docker compose creating our two data sets over there and now we're just going to load in the first data set over here. So we can take a look at that as it's actually working like it's g as it's working see if it's working. Cool. Okay, so some stuff is like really

11:23fast and some like it's it's interesting to see like how things are actually kind of working for all of that stuff as well. So we took our CSV, we just loaded it, we took a look at it, right? Not much is happening there. You can run the count on it as well to see like how many actual records we have. And we're going to see something around the lines of like 43 million, something like that. So it's going back and forth. It's processing. It's asking like the remote

11:46server for all this information. So it's having to aggregate all of that and then bring it back down. Remember this is CSV at this point so we don't actually have any statistics. So have all of that stuff. We're going to bring in the November data set as well. Do the same thing. Um, yay. Got to do the count on that one. And then we're going to actually go do some actual work. So so far all we've done is just brought in like the two CSV data sets. We've

12:09created data frames for them so that we can essentially kind of go and and do some work right over here. I want to bring in this e-commerce data set starting off with October. And so you'll see that we get a new parquet directory as it's starting to actually write. And so all of the stuff is like up in like running in our server environment. I know it's like a docker environment. So everything feels like it's all in the same place. That's

12:32for simplicity, but the processes that are running right now are essentially like the Jupyter environment and then like the server side, which is where all the work is actually happening right now. So this will run. I know it's not the most exciting job to watch in a live demo. um as our parquet is continuing to run and run and run and run and run. Um and so we'll just update this over here. We'll take a look at it and we'll see that we now have everything that we

12:55need. I can just scroll that over. So right now we have an entire month worth of data for that. I'm going to kick this one off anyways just because why not? And this will just give us all of November as we kind of go down. So this is this is a a Kaggle data set that has e-commerce data from 2019 and it's got two months of like uh view data and a bunch of other cool stuff as well. Um, so if you've ever been interested in like retail data, uh, it's a really fun

13:18data set to play with. So as that one's going through, we're just going to go over here and wait for a second. Load up our park over here. We're going to do now is essentially just go and create like a delta table. I'm going to wait for this uh, operation to stop. I apologize for this taking a while. It's almost done though. Almost

13:40done. Come on, demo gods, make it happen faster. So if I had a if I had a bigger like so while this is running let's pop up localhost. So we'll see that like spark uh spark connect is running. We've got a session my user is nobody because I'm running locally as a user called nobody and I have no user identifier for that. But we can see like essentially like every single one of our operations

14:02the statement is essentially like our it's like it's the parsed view of what what we sent as like our like our logical plans. Um, and that's been parsed and turned into an actual operation itself. So, this is now done. I'm just going to go through this again. So, now we have like two months worth of data. That's great. This is the first real test of actually using like Delta Connect. We should be able to create this table um and actually have it work.

14:25So, now that we have a data frame, we know that our table exists over there. You see that we have a new folder called Delta that has our ecom data and we have our staged commits over here. So, we know that we're using like we're we're using Delta 4. We've got a table like, you know, essentially set up with nothing in it. Um, and so now all we're going to do is essentially write all of our part data over to that delta table. So this is essentially just testing that everything is working as expected for

14:49um, let's go into our stages over here. So you can take a look, right? We're reading our files. We're mapping. We're mapping. We're mapping. These are all whole stage code gen. So we're not seeing like a lot of like the interesting like nuggets that are occurring right now. But if we go over into the data frame, hold on sec. Did it break? Let's run that again. We might have just hit it. It might snag or

15:11it might just be like off by one. Go, go, go. Okay, so right now, right now everything is being compressed as like Zstandard for all of our like our delta. So we see like a bunch of work right now uh going on on the left hand side. So we have essentially most of we're writing everything like in parallel. So we can just take a look to see how this thing is kind of growing

15:33out as it grows over time. This might be kind of boring. So it's just just running along which is great. Take a look at all of our jobs over here as well. So uh for every like every job itself, anything that we send over, so like every like every individual action, we can consider that as like each time we've essentially pressed play inside of a cell, we actually have a command

15:55that's been sent over. And so the command is also part of those plugins we initially looked at when we were taking a look at how to like set up like the server and like the client. And so, um, I think this is almost done. Cool. See how we're doing. Cool. I'm happy with that. Looks like we got everything now. All right. So, the last thing over there. So, we just finished a bit essentially writing. This is what we showed like in the demo where

16:17it's essentially we're writing our delta. We've got our path. We've got our source paret. The source parquet is essentially just a normal uh command to be able to read and write paret that Spark Connect already knew. And the only difference is essentially the delta format which came in through that plugin. So now if I want to read from my delta table like I would any other delta table, if I want to take a look at dates between like the 6th and the 16th and I

16:39want to take a look at only views and I want to order by ascending, I can now do that. Running it all on the server and then all I'm doing is getting back essentially just a buffer that can be essentially read back into a dataf frame locally. So that thin client all it's actually doing is essentially it's like if it's like if I could speak dataf frame, right? It's probably a better way of talking about that, but that's essentially all it does is it takes a

17:02dataf frame, it knows how to read and write from a dataf frame. It's just asking somebody else for all the help. Um, and so with that, that's pretty much the demo. And I'm going to kick it back over to you, Stephanie. >> And here you go. Thank you, Scott. I think gods were with you almost. >> Slight one slight problem. >> And here you go again with liquid clustering. It's one of the my favorite

17:25topic. I just want to make sure that if we have new joiners, they might want to know about liquid clustering. This is a really modern uh modern way of efficiently organizing um data. So rather than having to uh partition them or like order them physically in the in the storage, you rather organize them

17:48logically thanks to liquid clustering. So I'm pretty sure you're familiar with the first uh with the first SQL query. So you see that you should you can create your delta table and then you specify partition by I don't know uh it can be for this case type or value with clustering it's almost similar you just need to replace partition by by cluster

18:12by and that's it. It's very easy sorry and it would be super important why that's what we're going to see right now. So basically when you want to modify a partition uh you would need to rewrite the entire uh delta table which means if I don't know you have a table with a size of one terabyte you would need to rewrite the entire table which can be time consuming and can be a

18:35little bit also expensive. While for the clustering you just need to alter the table as you can see alter table and then cluster by and you specify the new value. Of course, you can go up to four uh clustering keys or columns. And this is very efficient. Why? Because when you run the optimize, it will run

18:59only on incremental data. And this is very important. Why? Because now if you go with partitions when you run the optimize it will run on the entire data set including the files that were already optimized unless you specify for example a work close that should be available for part for partition tables but again it will only work if you have

19:21for example date because you can specify for example optimize table customer where date is higher than I don't know 15 of July 2020 22 while for liquid It just if you if you added one row, if you run optimize, you will of course optimize only this additional row. And if for somehow a reason you need to

19:45reccluster the entire data set, you can just write down the optimize your the table name and then add the command full. It will reccluster the entire data set. But again, I won't recommend you to run it often. Like run it only if you see that you did recluster the your tables so many times and it may might have impacted the table then you can run

20:08the full and I guess you may have a question if I'm partitioning my tables how can I move to liquid clustering. Well, if your table is already partitioned you would need to rewrite the entire table. You cannot just move it to the clustering. And if you are already like as you mentioned using the clustering plus the zorder command you

20:31can reuse the same keys as that would become now your clustering keys rather than the partitioning keys plus the zordering. So this is very important. You can also if you already use what we call generated columns you can also use them. And of course the same as I mentioned hive hive partitioning that's what I mentioned plus zording. So it's almost the same thing but you gain you

20:54have more additional features. Just keep in mind when you use when you use the CL clustering you're going to automatically activate upgrade to the more recent protocol reader and writer and it will automatically enable deletion vectors column mapping and row tracking if I'm not wrong. So basically deletion vectors

21:16will help you avoid rewriting every time the park files. That's for deletion vectors. For column mapping, it allows you to have space or special characters in the name of the columns. And then row track. It avoids concurrent queries because for partitioning you avoid the concurrency writing because you can write to multiple multiple partition at the same time. While for

21:39the liquid clustering since it's not let's say partition but it's logically row row row tracking can help you avoid those uh conflicts. I don't know Scott if you have anything else to add. >> No, I think I think the big thing is that like for unoptimized tables moving to liquid clustering is smarter. I was also thinking too the demo that I was showing was all essentially just using

22:02high style partitioning and it'd be interesting to basically go and modify that to show what it looks like with liquid clustering as well. Um and I'm going to do that after the session's over. So be interesting to see if the how the performance looks after doing like those optimizations as well. I mean it's local so it's like performance will be good. >> Yeah. And I think the next one is called collations and this one is pretty new

22:26again part of delta 4 and basically it does introduce collation support allowing user to specify how string values can be ordered and of course when I'm talking this you will use it to compare the data and this also will improve sorting and searching within the data and as you can see in the example

22:48so I have a bunch of names some of them starting with capital letters, others not. And then I asked like, hey, order this table by first name. And as you can see, I have Aaron twice. The first one is with capital letter, the second one with not that. And as you can see, it did it did order them, but not as I was expecting. So with collations as you can

23:10see after now that I ask like order by first name you can see that Aaron are both ordered in the same or let's say cluster which can help me for example in case I have uh customers sometimes I I add my name with a capital letters sometimes not I can order them maybe and reduce or whatever other things that can

23:32be used and also used for people who may I don't know if you are there using data warehouse moving their data to a data lake to delta plus open source spar they can they can use it easily and how I can uh enable collations as you as you need to know by default when you create a delta table it's UTF binary and my recommendation and this is

23:56important make sure to use it with spark fork plus because it will have let's say more accuracy and trust me this feature is amazing why because you can enable it at the table level. For example, you can define that all the columns within this table will have a default collation which is UTF8 lowerase or if you for example with

24:20within the same table you can have maybe multiple uh multiple options you can define it at the the column level. If the table has already been created, you can just use an alter column alter table alter column specify the name of the column then just add the collate at the end and with the with the with

24:42the with the code for example in my case here it's uni code under CI and AI or if you are creating a brand new table you can define it at the creation as as you can see I add I added here the collections for three different columns the first one UTF8 lower case. The second one is uni codeci and the third one is UTF8 lowerase R trim. So you have

25:07all the options to make your life uh easier basically. Anything else Scott on your side? >> No, I think it's interesting that this feature itself is something that is available and has like it has been available in like kind of like traditional like OOLTP databases for a long time. But it's nice to be able to

25:29see like the performance benefits of being able to do this. Especially for as you were talking about like for sorting itself. So versus just taking a look at like what does like the binary representation look like and sorting based on the bin. It's much cooler to be able to see like this occurring to see like the error eron you know all within that same little chunk which is what you'd want to see as well if you're searching you know for a lot of matches

25:51like needle in the in a haststack style. So thank you. It's cool. >> Yeah. And I think the next feature is also very interesting and I think it's very underrated uh because this type widening which will allow you to change uh columns to a wider uh data type for example going from from in to long or double without

26:14rewriting the underlying with the underlying files. As long as you have of course it will allow you to have this schema evolution and make it seamless and efficient. And again, it's super important because the goal is not to rewrite the data. Sometimes the types may change and you don't want to rewrite the I don't know those two terabytes data. And that's why we talked

26:36first about liquid clustering giving you the possibility to change clustering keys easily. Now we're also giving you the possibility to change the column type without rewriting again this data. And basically this type winding will let you like do this schema transition to a broader type and this is important ensuring that the existing data will

26:58remains readable and the new data will follow the expected type and how will it work. So basically when you have this type widening the table records the new type and this is important but the past written data will still remain as is and this is very very important but there is also something very important delta

27:21clients read data using the whiten will the w the widening type I'm always struggling to pronounce widen will handle the conversion automatically and we will see some examples over here. So what I did I created the same table twice. The first one I did enable type widening and for the second one I disabled it at the creation and what I

27:44did afterwards was now that I created this table what if I want to switch the uh the uh the ID that I was integer to a double. So once I did the alter the first one where I have the type widening enabled it was a success. While for the second one it was a failure because I had to I have to either enable type widening or rewrite the entire the

28:08entire table. And also something to keep in mind and I think this is very important. If you created a table with for example ID integer and somehow you wanted to you inserted into this id this column for example a float it will automatically trim and keep only the the integer part which means if you inserted 2.5 it will it will always keep

28:31two it will remove those the 0.5 and you won't have a failure. So this is something to keep in mind else you can go with the u with altering the column and that's not the only thing type widening exists since spark spark delta 3.2 but as you can see with delta 4 we added more possibilities and as you can see bite can now switch to short integer

28:56long decimal and double short to integer long. So we gave you now more more opportunities and also for the date you can switch it to time stamp NTZ which is also spark 4 compatible. So that's why I mentioned most of the things should be should work fine with uh with with spark 4. And you're going to ask me a question okay should I always go and alter the

29:21table to change the uh the value or the type of the code. So yes and no. For example, when you do the insert or the merge schema depending whether you're using Python or SQL or what whatever language you will use. So it will automatically convert this column but we have some exceptions. So for example integer cannot go to a decimal or or a

29:45double and same thing for merge into. So those are the same condition. For example, merge into if you're specific specifying a condition merge into a column and where merge into a table from a view or another table, you're going to specify a condition. Once this condition is matched, you're going to merge into this specific table. So again, it will automatically take in consideration the new column type but not uh not convert

30:10from uh for example integer to decimal or double. So those are the exceptions. If you want to be sure, just go and do alter table and then alter the specific column and switch it to whatever type you want to be moving to. Just keep in mind that it it has to follow those specific source. So if it's not here, then of course you're going to have an issue. And something I've seen

30:35people asking about, okay, what if I I activated type widening? I want to disable it. So basically you can disable it with as as I did show you at the beginning delta. Uh typewiden equal to false but it won't remove it completely from uh from your table. And that's why if you want to really get rid of this uh

30:58this feature but again for some reason maybe your protocol reader or writer does not support this feature and you're blocked either improve your reader and writer or you can drop this feature by altering the table the name of the table drop feature type winding and truncate history and this is how you can get rid of the enablement of this feature else

31:22it will just disable it for the for the new new columns and the existing one will all will will still be white. >> With that like with that in mind of being able to like go and drop this feature. If you dropped the feature and you had a table that had an int and then you manually altered the table to allow it to be a decimal type and then new

31:44data was added prior to dropping the feature. What does the state of the table look like at that point in time? >> Sorry, can you repeat a question? I did the blog. >> No, no, no, it's okay. Um so if I if I have a table and say like the first 10 transactions have ID field as an integer for like the 11th uh transaction I'm doing an alter table and so I'm going to change like the schema declaratively and

32:07I'm going to go and change it you know it's like the manual operation to go and move it to a decimal type and then I'm going to go and actually drop the feature uh after writing in you know say like transactions 12 through 20. So essentially it's like you know it's integer split point decimal. So my table now has like essentially both types in it. What does it look like once I've truncated the history? Does it just

32:30still continue to show decimal and behind the scenes we're essentially just doing the transformations for the first 10 transactions. >> So basically when you truncate the history it will it will stop it will keep it will keep with the feature called checkpoint. it will keep the existing if I read this data I will still read it as is but the new one will

32:54still will be back to the existing one which which is the integer if I'm not wrong it was it was an integer >> okay so it's just going to sit there and it's going to truncate back this is it goes back to your original okay >> so just going to truncate everything else after the decimal perfect >> exactly yeah and also the next feature I think is also very important specifically for those who have to

33:17handle semistructured data. So basically over here basically it does enables flexible and efficient storage of this semiructured data and it will make it super or much more performant than traditional approach of storing it as a string and then having to trim it or do a lot of transformation. So basically in the past you had to create for example

33:39JSON example it would be an ID and then your ch your JSON will be stored in the data as a string. While now you can just create a type called variant and why variant because basically it means that this this file or this data will keep switching because you don't know how it will looks like. So that I think that that's why it was called par and it's

34:03why it's important because for three reason you know it has a native service where it stores data in a binary strongly uh type format enabling efficient storage indexing and query performance and in contrast string columns require as much parsing to JSON text at query time which make things

34:26slower and a very in in this uh resource intensive and the other the other of the second option is because of this schema evolution and type safety. So with variant you can directly access nested fields using the uh dot annotation that's what that's what I will show you delta x understand the structure and the the types within

34:49>> hey real quick um I think the microphone is running out of battery potentially it sounds kind of like uh like a windy day um just wanted to bring that up real quick sounding a lot better right now >> uh so uh yeah so I said this reduced the the errors and making queries more robust. While for the string approach,

35:11it lacks types type safety and is prone to it error pron. So that's and so that's why varants also important. And the last one is about performance. So when you query uh when you query variant those column when you query variant columns this one are faster and more expressive and they support advanced analytics. You can use them for example

35:33with the with the with BI tools. While for the second option, it requires like repeating parsing and makes like some expressions a little bit complex. So that's why you can see in the example before we need to get JSON and specify the object. While for the second one I'm just as I mentioned because variant was a data data type. So now I just access the name and I name it name and

35:57then age and then if it's nested address city and so on. And you can see like you can see a huge improvement in the performance. And how can I enable it for existing column? Very easy. Just alter table and you need to set uh the like the uh this command delta delta.feature variant type preview supported. Just

36:21keep in mind and this is a warning once you enable this feature. This table won't be readable if your client does not support variant data type. As of all the other features I mentioned, it's dependent to the read the data protocol reader and writer. So the compatibility is required and variant columns cannot hold data larger than 16 megabytes for now or as of now. Maybe things will

36:45change in the future and I'm not saying the entire column but just like for one row it can't be higher than 16 megabytes but your table can be two terabytes. There is no no problem. Uh anything else to add Scott over here? Yeah, I was just going to say that like for for a lot of a lot of people who are testing um testing ingestion of like new data

37:09types and they're not essentially sure of what things should look like yet. Like a lot of times like when you're going and you're prototyping a lot of things, things changed often and variant type is a very good way of being able to ensure that that data can mutate in a way that's still performant. And that's like one of the things I found interesting about like variant type in general like a lot of like the use cases

37:31are like you know IoT related and it's like you know you have one sensor but now you want to have essentially like an aggregate of multiple sensors to you know have a more performant like uh read theut for like an individual table and to do that in the past you'd have to join a bunch of different data across a bunch of different join operations to create a new table or you'd create like a materialized like synthetic table of all the other tables or with variant you

37:54can basically just stuff new data into the variant type itself. Um, which is an interesting way of being able to like, you know, modify and mutate data in a way where it's still performant, which is really interesting to me. >> I think I've seen a question uh does the variant type requires any upgrades at the reader level? Yes. So the variant uh

38:17feature in Delta Lake requires upgrading the reader protocol which means uh all the readers not just writer they must support this this feature to read uh tables that's v that has variant type and I think the last one the last feature I wanted to mention uh it's called checkpoint protection so delta 4 introduced this protection which is a

38:41feature that that ensures that your table remains safe and it's still compatible even when you uh you drop some features by dropping features you remember I mentioned you can drop maybe deletion vectors you may drop type widening so the goal is goal here is whenever you drop a feature delta lake will automatically rewrite the recent

39:03table history and protect the checkpoint and which means even if I query this table it will still be let's say will still function because in the past if you drop some features it may cause some some issues. So this one will pro will prevent the protocol downgrades downgrades from being let's say accidentally removed during those cleanup operation which means and at the

39:28same time it ensures that the older clients can still reliably read this history to be able to populate back the this table and the goal it says and as Scott mentioned during the like the question he asked at the beginning of the type winding when I said you can drop it. So it does protect the checkpoint mark at the transition point which means older client can still still

39:52can read up up to this the checkpoint and newer client will handle the rest. So that's how we can make sure that it's still compatible and this one is uh part of uh Delta Lake 4.0. You can can find more details in the um in the Delta Lake documentation, but you can also access the um the uh the code in um in GitHub.

40:15You can see the exact implementation and how does it work. So I I tried to simplify it, but it goes beyond what I what I've just explained easily right now. I think it's also the interesting thing with it too is that like if you think about creating a point of no return where you're like we're going to force upgrade a table at a certain point because I want to opt into all these new features by being able to allow an older

40:37client to read up to a certain point. It's like a safe cut off without essentially cutting like you're not just immediately blocking them, right? It's like, well, you can read up to this point in time before upgrading, which is kind of a nicity to do that, right? Where it's like, you don't have to upgrade today, but the second you hit this data, you'll have to upgrade. And that's cool because it's a way of communicating through the table to a

41:00customer about capabilities. Like, we're always upgrading things, right? From computers to cell phones and everything else in our lives. And like the same thing goes with the data, right? Technology improves for the right reason. And the hardest thing to do is to get people to actually upgrade. So it's like, well, here you go, right? It's like a line in the sand. So this is this is great to know as well. And there's probably a whole entire like subset of problems that like the

41:24checkpoint protection is solving for now as well outside of even just changing features in the table as well. So this is neat. Oh yeah. Uh I I'm going to sorry I'm like this is just a picture of me. So this uh so long vu who's in the picture with me from like the open lakehouse mini summit from back in June. I have mine too. I'm like, "Yay, marketing." >> Yeah. Thank you, Carly, for the wonderful glasses. And I think this one

41:47too, my Delta Lake cup. But Long Vu was originally the one who told me about like the new updates to Delta Connect and got me interested in like looking further into it. And so, we've been kind of chatting back and forth about like who in the community kind of helped put everything else together. And so, it's probably hard to see right now, but uh essentially everybody who contributed to Delta IV is in like the call out. And then also like a thanks to like Martin

42:09Grund who came up with Spark Connect originally and then also like I'm shouting out to Buff who helped create the tool chain that makes Preduff really work well within Spark Connect and like the Delta Connect uh environment as well. And outside of that I think we're going to move on to questions. >> Yeah. And I and just to answer I think someone asked a question what would happen with checkpoint detection when we change column mapping. Yeah. So you

42:33cannot drop all the features. You have the list of the uh of the features that you can drop using the trunk history. But it cannot be done for all all the features. And I got your point because if you drop for your column mapping then the name of the table the name of the column will be will be a problem because you cannot have this special characters. That's why you cannot drop all the

42:55features using the tranate uh feature. Yeah, I think so with that also for that question there's an extension of the question which like would we be able to revert to previous state like nothing happened because right now as far as I understand we're always getting some extra files under the hood that are not deleted if we revert there's probably like the additional like underscore metadata files and things like that

43:17which I I don't have a good answer for that. I haven't tested dropping all features yet, but there's probably things that might kick around and um if they if you're if you know that they're not needed, you can always go and delete them, but there is always a risk of corrupting the table. So, if the overhead's not too high for it, you could just keep them around even though it's kind of like, you know, it's

43:39keeping kind of trash around until you need it again. But I'll go over to the quick the actual Q&A as well, so then we can start to uh answer other questions as well if there are any other questions. Um, or I can just chat with Yousef about, you know, cool stuff. Like, how's Paris? >> Well, it's windy. Not that good. I haven't seen the sun for a very long time. >> It's sunny right now in California. If

44:01you want to come over here, jump on a plane. I'll host you. >> Yeah, next uh next June for Dice. >> Perfect. That sounds awesome. Well, if there's if there's no other questions um and people don't just want to hear us ramble, we can we can call it quits for the day. Thanks for coming. uh thanks for coming and for sitting through yet another like a high performance Delta webinar. If there's anything that you'd like to see coming up or other deep

44:23dives, we have a whole entire like fall full of interesting deep dives including a very deep dive into Delta Connect. And if you want to go into variant types or if you want to go into like type widening or anything else like that, we can commit an entire webinar to really anything. So just reach out to us either in the Slack channels, reach out to us through LinkedIn, etc. Anywhere where you see us online. And with that, thanks. Bye everyone.