Catalog Commits with Unity Catalog: Multi-Table ACID Transactions

0:04Hey folks, today I have the pleasure to host for the second time Scott Haines, but today is a special day because Scott is now a DevRel at Databricks. First of all, congratulations, Scott. >> Thank you, sir. >> Super excited to have you with us, and I think for those who didn't watch who watched the last two videos I I I

0:26published, I talked about the multi table transaction, and this feature relies on catalog based copy. It's mandatory. But this concept is pretty new to me, so can you walk us through this concept in Unity Catalog and maybe later for Delta tables as well. >> Yeah, for sure. Yeah, I mean this is So, if you think about the catalog, right?

0:48The catalog has been one thing that people have kind of fought over the years, whether it was Hive and having problems with Hive, whether it was like, you know, people kind of creating their own, you know, catalog catalog ask type system on top of like HDFS back in the day. And now we've kind of moved to like this, you know, kind of more of like a unified catalog like environment where you've got things like Unity Catalog that do all of your governance in like

1:11one spot. They manage like the ecosystem. They tell you where everything is, who has access, who shouldn't have access, what your columns are, why those columns should exist, why certain columns maybe don't need to exist because no one's using them, things like that. And so like with the catalog essentially becoming like more of like an engine on its own, you think about like the amount of effort it takes for people to say like, you know, I've adopted a specific open you know, open

1:33lakehouse spec, right? Whether it's Iceberg, it's Hudi, it's Delta. That in itself is difficult. Then it's like, okay, well we need an engine connector. This is a connector for Spark. This is a connector for Polars. This is a connector for XYZ other ecosystem. And that becomes a lot more difficult because we're taking all of like the onus on what should be correct, and instead of having like one place that's kind of the source of truth for all of

1:55that, we're instead saying, everybody figure it out, right? And that's a lot more effort for everybody to kind of maintain that. And then it also is problematic when you think about like moving, you know, large specification major versions, right? So, historically, right, it's like, you know, we're moving from Delta, you know, two to three to four, etc. Um but we also have with Iceberg, right, moving from the V2 spec to the V3 spec has caused that's been fairly difficult in the community. And

2:17then from V3 to V4, there's a lot more kind of work, you know, that is also then that's like, you know, it's like, "Oh, we just finished V3." Well, now we got V4 coming as well. And those types of things, you know, you're kind of always kind of chasing that next release that needs to happen. But, when you have something like, you know, the Delta kernel, for example, that's sort of the same sort of mental model, right? Where the kernel should take a lot of like the effort away from you as a, you know,

2:40engine author that's creating something that is interoperable with Delta. And now with the catalog-based commits, we have a different way of saying, "Well, this transaction completed correctly. I have recorded it, and now I'll make this new transaction visible to any reader." And that's something that I think is like a pretty cool thing, even if really, you know, not a lot of people actually do see that as well. And

3:02because it's going to all happen behind the scenes, right? You've connected to Unity, you've connected to like the Lakehouse ecosystem of your choice, and this is all just kind of magically occurring for you, if that makes sense. >> It's No, yeah, it's it's a it's clearer now. And is there any specific thing to enable? Because I I remember with multi multi with the multi-table transaction, it was a configuration in the table

3:25properties, like uh catalog-based commits enabled. Is it something different when it comes to Unity catalog? Or how like is What should I do to enable it? >> Yeah, so, I mean, with with Unity, so I have I have a demo I can show after um that has that kind of goes through the server properties. So, because this is still kind of an experimental feature, um you have to opt into this experimental feature as well in the

3:47server properties. And so, we can take a look at like the Boolean value for that as well. It's really as simple as going from a uh you know, it's like now it's true. Yay. Let's restart the server and now like we can we can enable this new feature. And then on top of that there's also a Delta feature that needs to be enabled as well so that the tables that have the new support for catalog managed have the ability to actually write. And

4:09so it kind of requires both sides being turned on, right? If I turn on just the Delta side and the catalog's not ready for it, then you know, that's sad, right? It's like bad news bears. And then on the other side, right? The same sort of thing as well. So essentially two things have to be opted in and then your tables no longer are writing only to the file system. They're writing first to staged commits asking the catalog whether or not it's okay to

4:31essentially insert at the specific commit ID. And so it's kind of like a change to the concurrency controls that have been in place for a while in the Delta ecosystem. But now the catalog is the one saying yes, you can do this or no, sorry, you have to try again. And so it's just a different kind of different way of like opting into concurrency controls. You kind of already talked about this like what is like the difference like between like Well, we've got up I guess two terms. And so we should really kind of talk about what

4:53those two terms mean, right? So you've got catalog managed tables when we're talking about how this new catalog managed commits works with a Delta table. And so Unity Catalog provides us with something that we're calling like CCV2 which is a specification for catalog managed commits. And then Delta has essentially opted into that spec and said, "Okay, I know how to I know how this works. This is great. I'm going to

5:15just change my way of writing tables." And so if we get anything else out of this whole entire podcast, it's that the catalog managed tables use catalog managed commits from Unity Catalog as in right now Delta Lake and Unity Catalog are essentially the two partners that are making all this stuff work together as well. Okay. So we've talked a little bit about this as well. And so pretty much all of this kind of you know, if

5:38you were to boil down this whole topic into what is this doing, it's all about catalog coordination. So the catalog now becomes the source of truth versus well, I wrote over here and you wrote over here. This should be in the correct order but we didn't have the right, you know, we didn't have the right concurrency control in place to make our rights, you know, work as expected, right? Things came in out of order. Now

6:00with the catalog essentially being like the atomic arbiter of how your commits are actually recorded, you don't have to have those types of problems anymore. And so really like you think about this too like there's, you know, there's three kind of main things, right? It's always discovery because we're talking about a catalog and the catalog is what allows you to see things. But also like in the past with a lot of Delta Lake tables, people used to be like, oh yeah, my table is at this S3 path. It's like

6:23that's really cool. What if you move it? What if there's a V2? What if something changes? And it's like well then well then you have to figure that out, right? And now with like the catalog based commits, you just have a catalog entry for a specific table. So say this is like your users table or this is, you know, an e-com orders table. The only real difference with all of this is that you now just have a table name. So it's really no different than having a managed table in Unity Catalog. Just the

6:45management and the coordination of your commits is now also, you know, that is part of what the catalog is now doing as well. >> And this is Yeah, sure. You go. And this is what will help for example if I'm using an external engine to be able to write into the managed Unity Catalog managed tables. >> Yep. Yeah, so everything So the commit API is also open as well. And so um if

7:07you're an engine creator that wants to do something similar as well um and work with the catalog managed commits, um then you can also extend the protocol as well. And so some of the things that people have wanted like in the past it's like well on commit, you know, I want more metadata, right? Who is the user? Why did they do this? And a lot of this stuff went into like the Delta specification as well under metadata, right? And so it's like well

7:29if you have metadata that's going through in one like in one path, is there a better way of doing it or a different way of doing that that's more interoperable, right? It's not necessarily tied directly to the Delta Lake specification but rather it's saying I want to commit this, you know, table of some sort. And now the catalog understands a little bit more about what what I'm attempting to do, why I'm doing it, you know, is this like a triage-based job? What else is this? And

7:52so like having kind of an extensible layer in the catalog makes that it makes sense. Versus having to make sure that every, you know, every single table specification is now opting in to do exactly the same thing. So, we're saving people time. Um >> That's right. That's our goal, right? Like fingers crossed, that's our goal. And then, you know, the last thing too about this whole entire change too is that, you know, for the longest time

8:14Delta was essentially a file-based protocol. I wrote to the file system. All commits and everything else were managed through the file system. Um and then that put a lot more kind of complication on the actual client to make sure that everything is coordinated as coordinated as as it needs to be for the table to be correct. And now we're essentially adopting the same design that's been popular on like the Iceberg side since Iceberg essentially came out

8:36with the Iceberg REST catalog. And so now that, you know, now both you know, you can do the basically the same thing on both sides, which is pretty good. So, you know, as we're kind of moving towards like these next generations of like all of our open lakehouse formats, um you know, the catalog is something that's going to be very much, you know, central to all of that. So, moving on. We talked a little bit about reads. And so this is kind of more of for people who are excited about what changes. And

9:00not really a lot changes, right? So, if you take a look at the left-hand side, this is a typical read with Delta. I have a reader. I say, I'm going to read the log. Where did I last read? Okay, great. I'm going to pick up from this checkpoint. What does that encapsulate? You know, what has occurred in what parquet file? So, I need to pull back in, right? And so if we take a look on the right-hand side, we've got like the log and we've got staged commits.

9:22We've got Unity Catalog. And so staged commits is essentially new to this whole entire like, you know, to the catalog managed commits. So, let me talk a little bit about that. Right now, we also shouldn't talk about that more when we go into the rights as well. Because the staged committer is part of the right path as well. So, if I'm a reader, I'm going to go and say, "Hey, Unity Catalog, I have this table. This is neat. I want to read from it.

9:44What can you tell me about it?" And it's a basically say, "Okay, you know, I'm going to be a pretty much I'm going to tell you exactly the same thing that Delta told you before. Only now I'm just you're pulling that information, that metadata, back from the UC. And so, Unity Catalog tells you exactly where the parquet files live, why you should care about them, and then, you know, is there any kind of like, you know, is there anything else you should know about this as well? And are there deletion vectors like, you know,

10:07involved, anything else like that?" And so, you have the ability to basically read through the kind of traditional path, like through the Delta log. And you also have the ability to read into the stage commits that haven't been essentially rectified into the actual like core Delta log as well. And so, you have this kind of two-phase commit path, which is interesting as well. Um, if that makes if that makes sense over there. The big thing is the catalog

10:30is responsible for like the commit visibility, and you shouldn't care essentially as like a reader of a of a table whether or not it's going to be correct. You know, Unity Catalog's responsibility is to make sure that the table is always in the correct order, and that the data is exactly what you believe it should be. And so, there's no more kind of guesswork. You know, it's like, "Oh, did we have the right you

10:52know I don't know. Were we using, you know, the back in the day there was like the S3 guard and other stuff like that?" And then there was like the other like was it Dynamo DB like a system I can't remember the name of it. I'm like the thing. >> Yeah. In the AWS it's required the Dynamo DB to be able to orchestrate writing from multiple engines. And I think with this with the catalog managed tables or

11:14catalog managed managed >> managed commits. >> commits >> Yeah. >> It will get It will It will get rid of this of the database. >> Yes. Yeah. Then Unity Catalog already has a database, and so that ends up being what's essentially doing all the work for you as well. And so, if about like killing two birds with one stone, which is kind of like a bad way of saying things, right? Cuz there's death involved. But if we think about this is

11:36like being something that's optimistic, right? We now do a lot more with less, um which is really good. And uh so we take a look at the rights, right? Rights before, as we talked about with like the Delta DB like committer, that is essentially the only thing that would help you to ensure that, you know, write one, write two happened in the correct order, you know? Cuz like this on the left-hand side is traditionally kind of like also like, you know, this is Delta.

11:59If we want to think about this as like, you know, Delta before, you know, uh the catalog managed commits, then this is exactly what we see on the left-hand side, right? Really nothing changes, but, you know, I, if I'm writing, and you, if you're writing, if you both if we both think that we're going to be essentially writing like, you know, the 13th offset, then, you know, that's great. If we, you know, if we do this incorrectly, then essentially we, you know, we need

12:22something that's going to be there to ensure that we're going to have some sort of blocking right. And so, if we take a look on the right-hand side as well, we still have the same interface interface that's there as well, right? Where it's like, if I'm going to go right, but I'm trying to overwrite what you wrote, then I'm going to have a failure. It's going to be like a concurrent modification type exception. That's great. Now Unity Catalog is going to be the one that's telling us whether or not that is, you know, if if I'm

12:45failing to write. Um versus us having to think about doing that in some other way, right? And so now there's one way to also think about how those concurrency controls are actually enforced as well. And so, if you think about like governance as well, this is just another form of governance, right? This is right governance within Unity Catalog as well. And so, you know, excuse me. I'm like like I was I promised not to

13:07cough cuz of allergies, but here we go. I'm doing it anyways, you know, it's a great Monday morning. So, right coordination occurs in the catalog. Something new that's been introduced is essentially staged commits. A staged commit uh for me occurs in my file system. It can also occur on like an S3 path. Depends on if you're writing in writing locally, if you're writing to, you know, a fully fledged, you know, environment. From the staged commits,

13:29I'm then asking Unity Catalog, take these staged commits and now go apply them. Uh and so, staged commits themselves become part of the Delta log over time, as well. And so, you don't have as many of like these like uh the temporary kind of missed commits that you would have in the past, where it's like, we tried to write, we failed to write, here's kind of like these like orphaned files that are sitting around in like S3. Now, the staged commits

13:52themselves become part of like the Delta log over time. That's all part of like the There's server properties for a lot of stuff and for that. >> All right. So, up next we're going to take a look at uh the demo environment, which is called Unity Catalog Playground. And so, without further ado, uh let me go and share the screen. All right. So, what we're looking at

14:15right here is the uh the main environment itself. Um and so, if I just get rid of the Docker Compose and the server properties for right now, uh let's just go render the read me. So, Unity Catalog Playground. Um so, this whole entire environment is essentially just an area where you can get comfortable um working with Unity Catalog locally. Uh and so, in this case, there's really

14:39two steps to get the whole entire environment up and running. The first is to run Docker build. So, let's open up a new terminal. So, new terminal. Get that up and up over on the bottom. Run the build. Make sure that Docker is running as well. Uh otherwise, this is going to fail cuz there's no Docker uh locally. So, if you don't have Docker

15:01installed, uh you're going to need Docker Desktop. Uh or for me, like I use Colima, uh which is a Docker alternative um as well. So, we've got everything built. Um so, then our next step is essentially to just Docker Compose up. So, while that's starting, we'll see that we have Unity Catalog and then we're on most park. Um and for right now, I'm just going to go and uh

15:24take this link. So, we'll follow this link. We're going to open it up. Um, and then we're going to get back to that in just a second. And so, the first thing though we want to do um First thing we want to do is actually just kind of go through and see what is it that we're actually doing here, right? So, it's great that we have the ability to run Unity Catalog locally,

15:46but what else is it that we want to do there, right? >> [snorts] >> What's actually running behind the scenes? So, right now we have this ETC conf, and in there we've got server properties, and these are for Unity Catalog. And so, uh we're talking essentially about the managed table features. So, on the server, uh managed table can be enabled. This is what gives us the ability to have catalog managed

16:08commits, uh which is what this whole entire uh podcast episode is all about anyways. And so, um as long as we have server managed table enabled, um then we're all good to go. So, now we're back, and we're taking a look at the uh at the end result of us essentially running Docker compose up. And so,

16:30we're running Unity Catalog in one container, uh the Unity Catalog server more specifically, and then in the other container, uh we're running Mirror Most Spark, and that's what's giving us this uh the UI itself. So, if I flip over to the UI and then take a look at like the actual catalog managed tables uh notebook itself, uh we'll see that it's essentially been mounted inside of the Mirror Most Spark uh container environment, uh at the path

16:52essentially notebooks Unity Catalog delta. uh dot pi. This is uh this is also the same path that you'll find inside of the actual repository itself, um just under like the Mirror uh playground notebooks Unity Catalog delta. Um, and we're running in Docker. We've covered the uh Unity Catalog uh server property server managed table enabled true, uh

17:14when we're kind of running through what was in the actual environment itself. So, I don't have to go into that too much more other than um to specify again that that's how we enable the catalog managed commits feature, uh which we're using on the Delta side as a catalog managed table. So, uh switching back down, uh we've got Delta uh version 4.1.0, Unity Catalog version 0.4.0. Uh this is

17:37for all of our assets that we're pulling into our Spark session itself. Uh and you also see that we have a Unity Catalog server URL. Um this is pointing locally inside of our Docker network. Um so, it's traversing the internal Docker network um to the host name Unity Catalog, which it so represents our Unity Catalog um single node server itself. Um

17:59the schema itself, we're going to actually use our own, um but I've set this up essentially like initially so you understand essentially like what we have, you know, what our capabilities are, right? So, the main thing that matters is that our Unity Catalog main catalog, we're calling it Unity in this case. So, this is what helps us actually configure our Spark session itself. Let's put that into memory and now move into the actual Spark configuration

18:22itself. Um so, we're bringing in the Delta Spark package, uh which is essentially the jars that make us, you know, make Spark extended to be able to work with Delta Lake itself. We're also bringing in the Unity Catalog jar itself, which will enable us to be able to uh interoperate with the actual Unity Catalog server itself. Um and that's the way that we're going to be able to do our catalog managed tables.

18:44Um so, you'll also see that we brought in our Spark Delta extension, the Delta Catalog, and then over here, we set Spark SQL catalog for Unity to Unity Catalog Spark. You see single catalog. Uh it's called single catalog because it's essentially representing everything within uh the nested tree that's underneath the Unity within our Unity Catalog. So, you think about this as like a, you know, say this is a

19:06developer catalog, um you'd start working with the schemas inside of that developer catalog, but you wouldn't have access to all the different top level catalogs themselves. Um so we're using the same URI URL that we used before. You'll see we have no token and that's because we're running locally and in terms of a governance, we don't need to have a strong position locally on our own governance um because

19:28this is also just for a demo. Um so kind of main tokens and other things like that can all be vended through Unity Catalog itself. Um and this is not going to be that demo where we showed. So also one last thing, we're defaulting our uh internal Spark uh catalog to Unity as well. Um that is in just in a way of simplifying, you know, the first catalog that's actually loaded into memory. So we got our Spark uh Spark config.

19:51We're going to pulling in all the configurations we have from the above cell and then you're creating your new Spark session. So this is going to load some jars, it might take a little bit of time. It wasn't that bad. So like 2 and 1/2 seconds or so, maybe three. Uh I didn't really read it that hard. Um okay. So now we have a Spark session which means that we have essentially like an entrance like an entry point to be able to now use the new catalog manage

20:14tables. Um but the other thing we need to is essentially like a reason to be writing these tables. Um so the reason that I created for this actual environment um is essentially like the notion of a pet rescue. Um so my wife and I love dogs. Um we fostered dogs in the past and we've been new members of like the SPCA um in California for a long time. Uh so in this case, we got a schema and

20:38I'm going to go create right now that's called sanctuary. Um there's nothing nothing too fancy here. We're essentially just creating an entry within Unity Catalog. And then we're going to essentially go and basically describe that schema itself. So you'll see that the catalog is Unity and sanctuary is that actual like internal schema namespace. So if we think about this as as part of like our three-part namespace in Unity Catalog, it's Unity, sanctuary and now we're going to create a table called

21:00pets. So in order to do that, we want to create a couple of those helper functions themselves. Um you can take more time with this uh, if you go into the actual uh uh, you know, environment itself. You can take a look through the notebook code itself, but uh, really all we're doing is essentially creating a method that takes a data frame, returns a DDL uh, from that as well. So, you take a table name, the schema, which is actually the struct type of the data

21:23frame, and then any kind of properties that you want to add to the table itself. Uh, it gives us two different ways of creating tables. So, let's go and load that into memory. And now we'll move over to essentially creating like a pet generator. Um, so we're creating a data class called pets. We've got a pet UUID, it's got a name, it's got an age, and a flag to say whether or not it's been adopted. Um, we would love all the dogs to be

21:46adopt- adopted themselves, but sometimes we've got some dogs that will hang out, and then we're going to bring those to the foster families, etc. So, let's load this into memory. And now we have the ability to go and create pets. And then also you can take it uh, there's basically a list of our Python pets, and we'll create a data frame from that as well. And so, all that stuff has been wrapped up nicely, so you don't have to worry about it. So, let's go create some pets. So, we

22:09generated some pets. Um, this is really no different than how uh, things are in the wild. People will get pets, people have a reason to be able, you know, to basically surrender a pet that then gets picked up by a rescue, and those pets then become pets for other people as well as they go through the adoption process. So, a litter one, we're just calling it. So, if you think about 101 Dalmatians or anything else like that, right? Or if you've ever worked, you've

22:31ever kind of worked with dogs. If you've ever had a dog before in the dogs department, they have litters. And so, litters are more than one uh, dog, and in this case we're creating you know, more than one uh, here as well. So, but we're taking the first litter. This is going to be like, you know, the first set of dogs that we bring into our sanctuary. And so, we're going to go and basically create a data frame. We're going to take that schema,

22:53and then we're adding the actual feature itself. So, uh, Delta feature catalog managed support. And then the UC schema is sanctuary, UC table is pets. And so, the DDL itself, when you go create that using the code that we did before, uh and now we can go and print that DDL. And so, I realize I've run through this before, so like it's like there's no

23:15real surprise as we're kind of going through this as well, but uh when we print the DDL, we get a create table if not exists sanctuary pets. So, it has basically the same schema as our data frame for our pets themselves, and our Delta table property. So, now I'm going to go and create uh that first uh we can go and create that table using our new method. Um and then, if I again, I've already

23:37run this, so you have the output, but uh when I see an empty data frame, um like seeing res show doesn't really mean anything. It essentially means that the create was successful. Uh so, you can go and basically clear that output to see it again. Um okay, now we're going to go and write this table. So, if we go and run this, uh we're running we're running. We've written our table. So, now we can go and see what those pets look like, right?

24:00So, the pets are going to be kind of, you know, they're going to be randomly generated every single time, and create as many pets as you want. If you want to extend the pet generator to add more names, um then you can do that as well. Um Sorry about that. Let me fix my mic. All right, so Spark SQL select

24:22everything we have from sanctuary pets. We've seen those. So, now what we're going to do is we're going to take So, in the pet generator, I created what's called like an an iterator, and we're essentially taking groups of that a grouped iterator. And so, if I think about this is like, you know, we've got 100 pets we want to then create uh batches of 10 pets, and then we want to write those into Delta. This is essentially just so we can create a number of transactions. So, if

24:44we go run this, we're going to basically go take a we take a look at each of the individual letters that we have in our pets iterator, and we're going to write those to getting the catalog. So, we're running those batches batch by by batch, we'll see this there in the top as well. Um so, if that little hourglass is moving, it means things are still happening. And you'll see we have 100 pets. So,

25:07this is cool. Um and so, like before we kind of move forward, um this doesn't feel that different from, you know, how we worked with Delta Lake before. Really, the only big difference is that we now also have Unity Catalog represented essentially as our, you know, commit arbiter. Um and so, anytime that we go and say I want to go add, you know, another batch

25:30of pets, I'm going to stage those commits, then ask Unity Catalog whether or not I can now still write those in the position that I expected. So, you know, say this is batch 10, I expect it to be batch 10, then, you know, Unity Catalog will tell me if my concurrency, you know, if my numbers locally are correct for uh for concurrency controls. Um and then if you want to be able to check and inspect any of the information

25:52on like the table itself, um you can use the Spark APIs, um but you can also use like the actual Unity Catalog client itself. And so, I've added a couple um essentially the input output um to be able to go run more queries to find more metadata about this table itself. And so, if I go and take a look at this table, which is the Unity Sanctuary pets, um we'll see that it's managed Delta.

26:15It's got a storage location um that's right now inside of our Unity Catalog table. Um and then underneath that, we'll see all of the different Delta features and properties that are enabled on the actual table itself. Um and you'll notice that we were essentially we set one property and then the catalog said, "Oh, these are the other properties that are important." And I think this is actually like a really interesting kind of, you know, power move made by Unity Catalog and

26:39this feature itself. Um so, no longer do you have to make sure that everybody is doing exactly the right thing. Um they have to bring um you know, a minimal subset of what they want to the table as they're creating new tables. So, the Catalog will essentially add the additional features that are required for that table to work by Unity Catalog. And so, this is like a big big lift for anybody who's worked at Delta before.

27:03Uh okay. So, outside of that, if we want to go and get like the table ID, storage location, these are required to be able to take a look at our commit history. And so, we can get all of that just by using the different APIs that we talked about about before using the Unity Unity Catalog client. Um so, take a look at that stuff over here, right? Um so, all the information we need um about our table itself is all located in

27:27here as well. Big one just being essentially the table ID and that storage location. So, from there, we can then hit this curl. So, this is still like a preview uh function feature, um the Unity Catalog APIs and like the CLI itself will be um providing, you know, more and more details later as we go. But, if I run this curl, um I can get

27:50back essentially like the commit version, timestamp, all the information that I I would need about the table. Um and all of these this transaction information is essentially what helps me to understand essentially, you know, what's my local position, um can I write or has that write essentially been written by a different writer and therefore I have to now get like the next identifier and things like that. So, consider consider

28:13concurrency controls are now um all essentially delegated to Unity Catalog itself. Um and so, if we take a look, we'll see that we have essentially a table itself. We've got some staged commits. Um and then we have commit history as well. Um once that's actually been reconciled uh with the Unity Catalog itself. Um and so, aside from that, let's take a look as well at just like the history of

28:36the table. Um we'll extend this a little bit more so we can actually see our create. And so, we'll see we had our create table. We had that later one and then we ran through the iterator to create a couple more batches. And so, the Delta history function from Delta table builder, all that still works. Um so, if you ever use like a Delta table API, uh you're able to still read the history of the table just like you would

28:59before. Um and then you're also able to read the same information from this uh the commit API endpoint as well. So, you have two ways of kind of looking at the history of the table itself. Um and that's pretty much it for now. Um so, thank you so much for tuning in and um you know, we'll be bringing more and more Unity Catalog updates to you as we go. So, thank you.

29:22>> In Lake House >> Databricks [music] Lake flow streams the flow. Iceberg and Delta Lake tables are glow. Unity Catalog governs Lake base OLTP base strong. [music] Genie takes to SQL MOSEY AND KAYAK PROLONGS. YOU sip and query and co-host like the fire. Data dreams scale higher and higher. [music] Next gen, drop it.