Delta Lake 4.0: Liquid Clustering & Cross-Format Evolution

0:00exabyte of data processed every day with Databricks which like sounds like it sounds insane to me like it's don't ask me how I know about baby shark the funny is even I know about baby Shar who doesn't know about baby Shar testing happening that ensures we don't lose one backward compatibility hey so today I'm with youf and we're going to talk about Delta L

0:22with TD um so for those who don't know TD is Staff software engineer at dat bricks and I think one of the most AC contributor to Delta Lake um so today I think he worked on spark uh spark even as an Apache PMC you also help writing the book learning spark um and I think

0:44when I join the bricks uh you were focusing on spark streaming I remember you even gave us a training on that my journey in Delta Lake starts about about 5 years ago and uh where it starts is that there was a small team of people who were building structure streaming stream processing in Spar and I've been working on stream processing in the 5 years prior to Delta lck and some of the

1:07lessons we learned is that it's giving users nice apis and nice capabilities to uh process the data in a streaming fashion at a large scale with fall tolerance guarantees and stuff solves only half of the users's problem ultimately the processed data is only as good as the user's ability to use it

1:33with proper guarantees data quality guarantees asset guarantees and stuff so we realize that in order to make the user specifically data Engineers data scientists happy you need to be able to give high quality guarantees on the data which means that traditional uh

1:57data formats like park or CSV or Json or orc were insufficient because they don't give asset guarantees out of the box that's what led us to think that maybe we can do better maybe we need to uh provide the same level of guarantees as databases provide while being as scalable as data

2:21laks so something that can scale to terabytes petabytes of data even if it's in a single table uh but gives all the guarant that is expected out of a database on much smaller amount of data so that's what led to Delta Lake that let's take same Concepts as in a database here is a transaction log that database maintains except make it work in a way such that

2:45it works at a different scale in a much larger scale altoe and on different file systems in a distributed manner with optimistic concurrency and stuff so borrowing the same Concepts as databases and making it work in a different set did the magic so that's how Delta Lake came to be did you start working with a a specific customer I think I remember

3:08um some presentation was it with ail that we started the project something like that yes exactly so back in [Music] 2018 um this was in the data plus AI Summit uh we were talking about structured streaming and uh we were giving presentation of structured streaming and so this person Dom from

3:32Apple he's the a a very big shot security guy over there who is responsible for processing huge amounts of data center logs to identify security concerns and stuff and for them traditional systems were not scaling and so they Dom came and talked to us specifically Michael

3:55armrest uh and saying that this is a problem that we have we can process the data but we cannot consume the data fast enough at the scale that we need it to uh that's where the idea like okay we need to really solve this and and this is like uh Delta Lake lore like Michael armrest went on vacation in a couple of

4:18weeks uh wrote a prototype version of what is now Delta Lake and uh went we went ahead with it okay here's a prototype that can do the transaction log update with asset guarantees uh that scales because even the transaction log that can be very large and is processed in a distributed manner using spark and uh let's try it

4:42out and Dom was awesome enough to take our first version and actually run it at the scale that they have and it worked beautifully and with even that protype version was something that was much more scalable than any commercial system that they were using uh so and that's where

5:04it all started like so we really within a matter of like 6 months from the conception of the idea that let's start of the Prototype uh we were running Delta lake at at Apple in the scale of uh petabytes like hundreds of terabytes per day being

5:27consumed being app ended to a table of logs uh so that was amazing so it kudos to the folks who are really impassionate about solving real world challenges and are willing to take the effort to actually adopt and try out new stuff

5:51even if there are Kings and stuff try out new stuff and and take the leap of faith that actually moves the scale forward in like make reaches New Horizons in the data world of what was not possible before it is possible now yeah I and I saw the numbers they talked about at data nit like I think they they

6:13mentioned something like exabyte of data processed every day with their bricks which like sounds like it sound insane to me like it's it's so big it's crazy and I think TI gave a good example of a a song like how many times this song will uh need to play to be to have the same amount of uh of data I think it was Rahul who gave this example yes Rahul gave the example like I think like it

6:35was like to paraphrase him that if you hear if everybody in the world hears baby shark and he say that baby shark don't ask me how I know about s baby shark the funny thing is even I know about baby shark who doesn't know about baby shark but if everybody in the world 7 billion people play baby Shar it will still take like hundreds of years to

6:58reach that count that uh that many uh count of four exabytes per day which is a mindblowing scale at this point like six years since after conception of Delta Lake it's being used at this scale it just blows our mind every day every year there is like I 2x 3x growth in the amount of data

7:23being managed on with Delta L and I uh just something I wanted to to highlight like for those who are interest they can read the Delta L paper it's called the high performance acid table storage over Cloud object stores it explains like the concept of Delta L and how it was implemented so feel free to have a look at the paper and and T at

7:46this scale like I would be super scared to have some bugs and you like you just get some data you know issues and you won't be able to read the data anymore like it would scare me really like how does it work how do you do test everything multiple times what's the process behind the scene because it seems crazy yeah yeah these are these are very good questions yes we have a

8:09huge amount of uh process and testing uh while we evolve the Delta protocol so Delta protocol is the specification that defines that here is how you can store your data that will ensure that all readers will be able able to read readers who follow the

8:31protocol can read the data and so that protocol involves here is how you stored the data which is just park files and here is the format of the transaction log of how you maintain the metadata of the list of Park files that are present that that contain the data so evolving the protocol

8:53is quite a laborious process uh because we want to ensure that nothing no reader in the world uh that follows a protocol breaks in an unexpected way whenever there's a new change in protocol there is a proper process by which readers can adopt those changes and uh therefore uh

9:16evolve towards getting newer functionality and stuff and so the process of uh updating the protocol is uh there's a proper RFC process where we put out here is the changes we are proposing the community can give feedback on this protocol change will work in this way in different connectors for Delta or it may break other

9:39connectors Etc that feedback process and then we absorb that protocol as uh that that RFC as a part of the main protocol and throughout the process there's a lot of unit testing integration testing happening that ensures that anyone uh we don't lose one backward compatibility to we anyone migrating or upgrading or

10:04downgrading that protocol the new feature support in the protocol are not broken in either direction so there's a huge amount of months of testing and both unit testing integration testing testing with real world data that goes on behind the scenes before we release a new feature but but the cool thing is

10:28that uh that we that this does not reduce velocity we have till now over 15 plus protocol changes protocol version so while there is a very strict process we have streamlined the process in such a way that we can work at a high velocity despite that process and build

10:50cool features for the users to pick up those are the stuff that we were to really are excited about right now features that are going to come up in Delta 4 uh in uh later this year and stuff so yeah this is this is this is a hard problem but over the years we have really streamlined how to deal with this in the right way so that we don't break

11:12compatibility for all the users perfect and uh Delta Lake like 4.0 is the biggest release to dates in term of features uh for like reliability stability performance so we have bunch of announcement like liquid going ga uh spk connect like uniform going G we have bunch of other new features like Delta

11:34connect variant identity columns and we've noticed that a bunch of those features are also available in spark 4 so how those two like projects coordinated to work to develop those two features at the same time that's a very good question and okay so Delta 4 as it stands right now is in preview and that

11:57is because Delta 4 depends on spark 4 which is itself in preview so uh Delta 4 and Spark 4 we are aligning in such a way that all the new capabilities that are available in spark 4 will be usable in Delta 4 and we hope to release Delta 4 as soon as in in the Delta Community

12:21we hope to release Delta 4 as soon as spark 4 is released by The Spark community in fact we want to align with the spark community release timelines and co-develop the cool new features uh very much in aligned together because that's the only way like spark is one of the most um uh and

12:47this is I'm wearing my spark experience spark hat on uh uh is that spark is one of the most Innovative engines where there's constant amount of innovation going on not just in terms of performance but architectural changes cap new fundamental new capabilities and stuff so we in order for users using Delta on spark to give them the latest

13:10and greatest capabilities available in both spark and Delta we want to uh co-develop spark and uh Delta together and for that we actually in Delta cross compile Delta with spark nightly builds and that's how feature like variant which required which is a

13:32new data type altogether available in spark 4 or at least going to be officially available when spark 4 is officially G released and G that was first developed in spark but in Delta 4 we have already developed support for variant type across them because we C uh we can we cross build Delta against both

13:55spark Master nightly builds as well as the latest release spark spark 3.5 so that requires a lot of Machinery lot of uh effort from the Delta Community to actually maintain that kind of cross build but we do that the Delta Community does that in order to ensure that the Delta users get the latest and

14:17greatest available possible to be available in spark and Delta as soon as possible and sometime I feel like it's a bit confusing to know uh what's in Delta and and what's in the engines of for example if we take the viance so the variance that's the new way to analyze instructure instructor data like semi instructor data like Jon so that you can efficiently save and and read and query

14:40Jon what's so what's in Delta and and what's in spark like the yeah which goes away that's a that that's a very good question and it's often hard to distinguish um so for specifically for variant I I like to put it like this so variant is a fundamentally data type like you have uh int float double or binary array and

15:05stuff like that right variant is a completely new data type and unless the engine understands how to process deal with this data type like how do you do query planning with tables having this data type or even unless the engine understands how to process data of that data type no format can really take

15:28advantage of the data type so in sparkk we have added the spark Community has added support for the variant type itself on how do you encode Json data in a much better encoding format uh in let's say Park files with as a column as a binary column in park

15:51files that encoding decoding of semi-structured data is baked into Spark so you in sparkk you can create parket tables with variant type because Park is a native format present within spark in Delta we added the support for variant type on top of it so that um you can get

16:16the what the best of Delta which is all the asset guarantees Etc that parket table Nativity does not give asset guarantees time travel versioning Etc you can take advantage of variant type with within the context of the and advantages that Delta provides so Delta in addition also added support for variant type using the capability and

16:39the native variant type support in spark that's how I distinguish I would like I kind of distinguish that the engine needs to provide certain capabilities that the format can take advantage of if the format says so uh if the format cannot unilaterally say that

17:01I'm going to Define this new data type that no engine can read that is not usable for the user so both needs to the support for supporting a new format and the ability to store that format's data in a reliable way with asset guarantees needs to go hand in hand so it means like for example if Doug DB want to want to support variant they will be able to reuse the same I think it's an external

17:24library that we have in spark uh so they have to plug their library to read the variance with IND du DB and then just use the metadata the Delta canel and and that's it yeah yes so what that is eventually how it is going to happen and the varant is still very new and so what I am urging the folks in the

17:47community who built variant type is to release a uh binary codec written in Native or rust code that the Delta kernel rust can use and dug DB any native engine like dug DB can take a use that binary coding and to read or write

18:09variant types so that's how uh we are imagining that don't Delta kernel which I would love to talk more about Delta kernel is like the um uh the middle ground Library the middleware or the narrow waste where if Delta kernel adds support for any feature in Delta variant

18:31or otherwise that any protocol level feature in how you stored the data then other connectors should be automatically be able to use it without really having to worry about what uh the underlying protocol format Etc is yeah and the kernel is very important in the Delta for the Delta Community since everything is built on the Kernel to help the adoption with the upgrade of the

18:54protocol the reader and writer for the connector's compatibility so for everybody not knowing about the canel so the canel is like it's a library written in Rust uh that you can use and plug so do we have do we say that we have two version of the canel like one in Java and one in Rust is that yes is that correct yes exactly exactly so we are focusing on both the Java

19:17community and the native Community independently by building this Delta kernel library in both languages so that uh there are less interoperability concerns like the native and the python libraries in the world can read Delta tables via rust because rust and python interrupt very nicely but Java and native code doesn't interoperate that

19:41well so that's why we building in Java as well for the Sparks and trinos of the world the python um library is just like binder on top of the of the Ros canel and that's it so it makes it really easy to support new engine and and and develop new things with Delta even if you don't use Java exactly Mak yeah and T I'm curious about something so we did

20:05talk about spark connect and how it's decoupling the client from the server what about Delta connect very good question so for uh for those who are not familiar with What spark connect is spark connect is a fundamental architecture change that is happening in spark where there is a clear isolation

20:27between where you your business logic is running your business logic in Java is running in a separate process and Sparks driver the the the one that does the query planning Etc to prepare for distributed processing in workers and stuff where that is running are decoupled into separate processes so your business logic is running in a

20:50separate process talking via clear interface RPC calls to the spark driver running as a server and this isolation decoupling allows the business logic is not tightly dependent on exactly what version of spark Etc is running it so the server can be upgraded the entire spark cluster can be upgraded without

21:13your business application business logic application Etc having to be upgraded or modified or recompiled with new version of spark and stuff so this isolation is this new architecture that spark connect is going to provide um now but for this to work what the interface looks like is on the your

21:37business logic side you the your code let's say you're writing uh spark data frame code as you're writing as your spark data frame code runs it generates logical plans that is sent over the interface to spark server and the spark server knows how to run those logical plans it runs those logical plans

21:59distribut presing whatever spark magic goes on in the background and sends back the data the result of that query as Arrow batches uh for this to work both the Sparks client side running the business logic and the connect server side needs to understand the same logical plans or how

22:21to run uh The Logical how how so the server needs to understand how to process the plans that the client side has produced for Delta Delta is a very Advanced format does crazy additional magic than What spark does so Delta has its own logical plans that spark doesn't natively

22:44understand uh so to make Delta work on both client and the server side Delta also needs to be architected re architected such that there is a Delta library on the client side that produces the Delta's magical plans and then there's another Delta library on the server side that understands that magical plans and can work with the

23:06spark server to uh uh execute process those logical plans and return the results so this is a and this is how spark can often uh new domains new architecture changes in spark can affect Downstream systems and Delta is ahead in the game in terms of adop ing those

23:29architectural changes because we already have some support of Delta as part of Delta connect as part of the Delta 4 preview where we've designed this split architecture in Delta as well so that one spark connect is released formally with Delta 4 final release uh with spark for final release Delta 4 with that will also work nicely so I I think one of my

23:54favorite feature is uh liquid clustering just because walking with partitions uh has always been such a pain I mean I've seen so many customer shooting thems themselves in the foots you know with I've seen partition at the minutes uh like and it was destroying all the performan and stuff like that so yeah yes can you can you tell us how liquid

24:16is working what's the what's the magic behind liquid clustering yeah so this is liquid clustering my my the way I think about is yet another uh idea that we have just borrowed from the database literature databases in their closed proprietary formats and

24:39stuff support clustering data internally so databases manage their own data layout and stuff and the in in databases engine and the data have traditionally been very tightly coupled together so the data layout of how the data is laid out in the databases files are very tightly CED with the engine and they can cluster in such a way that engine can

25:01read very well very very quickly and stuff in this modern world of data lakes and lake houses the data format is not proprietary it is open like Delta Lake Park all of these any engine can read it so now the challenge becomes that databases didn't have to deal with traditionally is how do you do Define the layout of the data to

25:27be such that engines any engine can take advantage of it partitioning was a that the data community's initial attempt to do some sort of poor man clustering which worked pretty well but that is really old gen to get to the next level of

25:50performance where you're not uh restricted by the limits of partition the lack of flexibility that partitioning includes that's where liquid clustering comes into the picture so um basically the idea of liquid clustering is that you the data format should be able to specify itself that

26:13how my data in this Delta table is clustered uh and the and the user the engine or the user should be able to plug in their own custom clustering such that it is works best for them so that pluggability is point number one

26:35that we bring that liquid clustering bring brings in in the Delta protocol itself it we there's ability for to specify what clustering you want to use but then we don't want users to have to specify clustering all the time so what there is a default liquid clustering that ships with Delta OSS which does a

26:57reasonably good balance that works with all engines of providing uh a balance of good read performance without very high right or clustering overheads and stuff so a previous attempt of this was Z order where you can Z order uh by

27:23various different dimensions that are not fixed and you get but the problem is Z order was that you have to constantly cluster rewrite all the data in the table you know uh as as more data gets added you have to re recluster the data every time that is a very large right amplification so in this new default

27:47liquid clustering mode we have defined additional Concepts that prevent the higher right amplification things like Z cubes and stuff which limits right amplification that we Define like chunks of data like here is a 100 GB chunk of data that is already perfectly clustered uh that you don't need to recluster but you're not but that clustering unlike partitioning is not set in stone you can

28:10choose to recluster that with a different set of columns if your workloads have changed and you want a different kind of clustering but you don't have to and then uh so uh and as new data gets added only the new unclustered data will get getting re clustered thereby making sure that you're not constantly rewriting all amounts of data uh all the time so

28:35compared to partitioning you have the flexibility compared to Old Z order you have much lower amplification with with similarly good uh read performance because of the clustering and four if you choose you can plug in your own clustering logic as well that suits your benefits so maybe one last question what what do you think is going to be on

28:57Delta five um maybe more integration with uh Iceberg or something like that what's your what's your guess yeah that is a very good question I do think and this is a very important topic of discussion ongoing discussion right now is how do we

29:20evolve the community the whole data formats not just Delta but uh how do we work with the other format communities like Iceberg hoodie to so that we co-evolve to a better world where all these formats can interoperate better uh and that is because we have observed collectively that today the onus the

29:45burden of choosing a format is on the users the data Engineers data scientists they're trying to figure out okay should I use Delta should I use Iceberg should I use hoodie every format has their advant ages disadvantages Etc but honestly at its core all of them are a bunch of metadata on parket files and

30:07now it's just a matter of different way of managing that metadata or different way of adding capabilities on top of the raw Park files and stuff and So based on this observation last year in Delta 3 we proposed Delta uniform which is our it was our first attempt to take that burden away from the users because

30:31in Delta we said that in Delta will make sure to generate metadata for other formats on top on top of the same underlying Park file so that the same table you can read not just as Delta but as Iceberg or as as hoodie and the community the data Community has all

30:53contributed towards making that happen that's why uniform is GA right now there is both Iceberg support and Hoodie support but that is only step one step two is that what we don't want is all this support to be stuck in the lowest common denominator of what these formats

31:15support if it is just par files yes that all these format support but Advanced features like deletion vectors which is like Park file plus a bit map of which rows have been deleted so uh that Delta has similarly Iceberg has positional deletes hoodi has its own M version we want to we don't want the C

31:40common capabilities for the user to be stuck in the lowest common denor of p park files we want all the advanced features of each of these format to be also available to the user without the user having to choose any and trade off between them so what we are step two is what we are working towards uh working with other communities to

32:02evolve to together co-evolve such that users can take benefit of the more advanced versions of the format uh as well without things like deletion vectors and stuff so maybe that is what is the will the biggest achievement of Delta 5 is going to be is

32:24a better interoperability story across all the f formats that is not limited to just the lowest common denominator but something much much much better awesome yeah and I think with the acquisition of uh tabular that's really going to help uh and I hope we can stop the kind of war between the file format because like it's not it's not helping anyone um

32:47honestly feels like it's becoming commodity commodity and and everybody should be able to use whatever they want without fearing to you know having to do big migration and stuff like that yeah we are the best minds of Delta and Iceberg are working together right now to build that better world for our users yeah I love it yeah good good feature H it's going

33:09to be awesome thank you so much TD uh that was super nice lot of insights I feel like I did the worst intro ever so now I feel bad but thank you and and I hope we'll talk to you soon and and you know discuss maybe about Delta five I think technically if you wanted to introduce me it will take us like five minutes like writer of the

33:32SP like learning spark book writer of this develop work on spark Community Delta Community I think like you want to introduce properly we cast at least five minutes to do it you all are very very kind thank you very much this was a lovely lovely conversation thank you thank you thank you T bye-bye