Apache XTable: Write Once, Read in Delta, Iceberg, or Hoodie

0:00and all three communities are vibrant all three communities are growing very strong the French English with pararan partnership it's okay I think I was following try to make it easy to interoperate in all the query engins available on the market hey today I am with Quinton and

0:22Kyle Specialist of file format one house Apache ex Apache hoodie and I believe the first to send a commit on Unity catalog OSS Project based on Michelle so we had the talk with her I think uh last week and she mentioned that you were the first one building this uh demo in like

0:44two hours maybe quick turnar around on it yep that's that that's amazing and um can you introduce yourself first to to the community yeah thanks thanks for having me on the show as well my name is Kyle Weller I am head of product at a new company it's a startup called onehouse you can check us out at oneh house. I've been in the data space for

1:06about uh 10 years myself building different data platforms building different data products I was at Microsoft for a long time right before I joined one house I was also a product lead for Azure Databricks so very close to uh datab bricks friends there and I was on the Microsoft side of that uh uh first party relationship and when we first launched the product on Azure um up until I decided to jump ship and and

1:29start building this new company um uh one house so in the same space in the lake house space and um excited for for the future of what we can build all right amazing so today we want to focus on xtable and see you know how it's working what it does uh but maybe before we start on X table could you give us a recap of the main file format available

1:51here I know there is Delta Lake ice bear goodie can you share your you know your vision on that and and which one maybe walks forward yeah definitely and and this is kind of uh some of the motivation of why we created and invented um uh Apache X table um originally it was named one table and and the idea and premise um was because there are three very popular open-source

2:16um uh table formats out there there's Apache hoodie there's Delta Lake and there's Apache Iceberg and all three communities are vibrant all three communities are growing very strong and each project has um various uh Specialties or or features or areas where it shines and especially when you break it down into use cases and you

2:38think of just like take take a step back and think of just what is an open source community in general right it's a collection of people that are working on similar problems and trying to solve similar issues and so they build this project together you know hundreds of different companies collaborate and contribute to these projects all three of them and and they're drafting the

3:00project in a certain way to meet their needs and use cases so if you look across them um that's where you see some some deviations in how the projects work and operate and and what they solve for so um without going too into the weeds we can go there if you want of like the differences in in the table formats what we saw from a onehouse perspective because we have a goal to um try to

3:23build what we call a universal data lake house try to make it easy to interoperate in all the query engins available on the market and what we saw is uh uh many of our customers and many in the community were overwhelmingly asking for hey I need to use multiple table formats actually because of you know some of this stems from um some

3:45vendors only choose to support one format uh I think most vendors support all the formats or multiple formats but a couple only choose one um and because of this that kind of like pins a user um out there in the wild to like force their hand into a choice of just one format which they um and and in many

4:07cases they they didn't want to have this this Choice put before them or they chose a different format for for other purposes and so we had an overwhelming um um push from users on like hey can you help me use um uh hoodie and Iceberg or hoodie and Delta or Delta and Iceberg um and because I need to use things like Databricks and snowflake together in a

4:29combination um for for very good uh reasons or purposes so that's where we um first announced that we were going to build one table and we uh searched in the community to see if there were like-minded folks um who also felt this need and very quickly um both Microsoft and Google came to the table and they said hey we're really uh burning on the

4:52same need like all of our users inside um of our Cloud products they need this flexibility they need this interop ability so they came on board very quickly we co-launched the project in November with Microsoft and Google then we donated it to the Apache software Foundation um just a few months ago earlier this year um and yeah now it's a aache x table and the community is

5:16growing we're very excited about the momentum though and I've seen a very interesting code um in your in the ex Apache EXA web page choose openness not Storage Wars and I think this is this is a very very important code and that's that's it's following my question does the format matters these days yeah

5:40good question good question I think 2024 honestly I feel like is the era of interoperability um and of course I wear some goggles here based off of my position what what we're trying to build right but also seeing this validation and and movement from more for more folks even from you know Databricks right uh Databricks uh this year

6:03purchased tabular and also you saw Alig gosi on stage at the data and AI Summit there's a big focus of investment and area that way of like hey let's create I think one word that I like that it used was like a USB what what did he say USB of of table formats or or something like this right and so you know alongside Apache ex table there's also um um other other movements as well like

6:28adult like uniform and things that help uh uh go towards this mission of interoperability um so uh yeah I think you know it's still it's still very important to um understand what the table formats are doing under the hood because then you can can find the right match to your use cases of like hey which table format will give me an extra edge of cost savings or performance or

6:52these kind of things but at the end of the day you don't have to um because I've seen organization I don't know if it's the right word have like anxiety about choosing uh a table format and and what that does is it slows down Innovation because now these companies instead of just moving forward on their data platform build they'll carve out some like research project and spend

7:14some three months to build a benchmark to build a test and go compare all these things and then go battle it out you've got camps internally of some that like prefer one or some prefer the other and so it goes through this whole circus that can just be avoided you know of course still do your research still do your testing pick something that's good for you but also you don't have to be worried about like being locked in

7:37because it's flip of a switch you can switch to the other format so if you feel like down the road you made the wrong choice flip um immediately to other table format or even you can use multiple simultaneously now um with with something like a pach x table so I think that lowers that barrier of decision making um um and makes it so that uh uh

8:02uh it's much easier to mix and match um yeah sure so so I wanted to ask you so you're working for one house uh who is also the creator of like the original creator of apachi hoodi uh I feel like hoodi is maybe uh the less I would say understood or wellknown project of the three uh the three being Delta Lake

8:26Iceberg and oodi can you share a few more details on hoodi like how is it working at a high level and what's a good use case for hoodi compared to the others yeah great question yeah Kyle that's the discussion we had that fact that hood is not well known in Europe remember ah okay yes uhuh even uh different locals or regions um of course there's different uh distribution of communities across those as well so um

8:50yes Apache hoodie I'll I'll start with a baseline that um like if you look into any measurable statistics out there whether it's a contributor or um uh users or other things about that attempts to measure Community size or or growth or adoption uh hoodie's um size and adoption is is very great and

9:13even um um when it comes to like contributors and things there's even more there than there are even for things like iceberger or otherwise but besides that it's yeah go ahead Quinn no I I think at the beginning AWS started to build everything on UD if I if I remember correctly was like the main thing yeah historically um um it had a

9:36first mover Advantage because Apache hoodie was created in 2016 um and um so it it was pre-installed uh first in all AWS Services now Iceberg has been added since like 2020 or so um maybe 2021 actually I can't remember the dates uh for sure off hand um so yeah there's some um first mover positions inide

9:59there more to your question of like um what is it and what are some of the use cases inside there so I'll start with similarities first similarities are um very close in nature to what like Delta lake or Iceberg offer some that that foundational under the hood it's par files and there are like Delta lake has a transaction log Iceberg has manifest

10:21files and Apache hoodie has what's called a timeline uh on top and the goal and purpose these are all like different ways to describe similar metadata the goal and and purpose is to um provide asset transactional guarantees which all three projects offer um the ability to mutate that data on storage um it also

10:43provides Advanced statistics um uh around the files to make it so that queries can access that data much faster um and can really um um get into hey how do I skip past all these files I don't need for certain query types right um now where things start to uh differ or

11:06deviate is maybe when it comes to some um um Specialties that are around like more near realtime streaming um this is where Apache hoodie's uh Community focuses very hard on building because as we talked about earlier um the communities naturally grow around around these use cases so folks that are

11:28like-minded around like lower latency um um processing or NE realtime streaming uh those like-minded folks will come together and and um crank on those hard problems together so uh Apache hoodie from the very beginning if you look even into its like first commits or first release versions it had um these capabilities uh called like merge on

11:50read which um uh versus a style of writing called copy on write um which which most systems have used up until recently now um Delta likee has similar characteristics now Iceberg has similar characteristics um but hoodie's had these for for many many years um and and what it offers you is the ability to

12:12because how I mentioned these are parquet files under the hood now parquet files are column n-based storage which are well optimized for super fast querying and analytics um but they're also the most expensive to write um and they take the longest amount of time to commit to disk um and so what what a Pachi hoodi does is it it uh Blends both parket files and avo files which are row

12:36based it's a row based storage file format and um when when it does this so now it can absorb incoming data in in load latency and um you'll have parket files that we call the base files and it will use Avo for what we call these log files and so this is like new data that's incoming and now tools are given

13:00into the user's hands to help balance and choose when to take those row based files and merge them back into uh the Bas files the column or parquet files and what this allows the user to do is craft in balance for um how to choose your your right latency versus your read latency and balance that right latency

13:22versus right amplification and and some of these considerations that um help you fine-tune um especially around these ingestion paths so when it comes to low latency data ingestion um these are these are some characteristics that folks are interested in in trying to hoodie for and and when you say low latency can you give us a few example like what's what's low latency yeah um

13:46and I can and talk about it from um like real world perspective as well so um folks like uh a bite dance or or Tik Tok they have um uh models that are trained um of course it's like some of the largest data Lakes I've ever looked into or seen or um witnessed um as well but

14:08they will in in um near real time help to inform the next video prediction on your on your feed um and so that data has to move very fast to go and run through uh inference and have a return to build the next like four videos that are going to be Downstream your of your thumb um to predict or folks like

14:31um um Amazon for their package delivery uh uh system um they use Apache hoodie to help uh uh the routing um decisions for for the packages um so this again in in near real time um and this could be maybe more in the order of minutes I I would assume I don't know for sure um

14:54but in in that sense now they can um make uh decisions of which p packages should get on which trucks um to Route them efficiently uh through the delivery system um and also you've probably seen that experience maybe um uh when you're ordering an Amazon package and inside the app it will tell you how close the truck is or it's like hey the truck's like 20 stops away or 10 stops away and

15:18so like all those kind of systems are are closer there does that help on the um the ly question CL yeah yeah yeah I think so yeah and and do you still have because I use H was a long time ago do you still have like an um online table which is on EO and then the uh like the Baseline table on packet and then you so you see the two tables in your catalog is that how it's working

15:41yeah great question um uh yes so both of those uh file formats will exist simultaneously and then what you describe is actually an interesting concept because we allow the user to query the table in different ways and so they will um like inside of a catalog um and so if you're doing this on AWS like Apache hoodie has like a catalog synchronization utility so if you

16:04synchronize these to glue hoodie can register actually technically two metadata tables inside the catalog and what these are um these two tables will point to um the actual files in different ways so if you want a query to get the most upto-date data possible and you're willing to pay a little bit of um

16:27extra cost to scan through those um uh log files those those aob based files this is what is called merge on read right the name and and term for merge on reads because during the read operation you're merging those on the Fly um or you can query the table and say hey ignore the latest base files I want my query to be as fast as possible

16:50um so we call these like read optimized tables so like ignore those base those um log files only go after the base files and so now someone can craft a balance right here where um you know when people start designing Lambda architectures they'll have like hey this is my hot path I pay more for my hot path because it serves these kind of purposes and so they might have a

17:12downstream like Flink reader or something like that that they point on that hot path that does the merge on read characteristic but other folks maybe it's for some bi dashboard that people only use every once in a while and it's okay for like 1 hour latency and if you've tuned hoody's compaction Windows to be uh shorter than that 1 hour Lany then they never even see a gap

17:36of upto-date data they just get the fastest queries possible um so it lets you like balance these together yeah and uh if I'm not wrong with in Del Delta switch from the copy on right to merge on read with the introduction of deletion vectors deletion vectors yes yeah and so just to summarize what you have said said so far

17:59um we have three formats Delta Iceberg hoodie all of them are Park plus some metadata except hoodie which ches the Avro but the main question is how does ex stable work and before that uh the previous name of ex was one table can you tell us a bit the story behind the

18:22chain change of the name yeah definitely yeah yeah so one table was the original name um for the project and that's how it was when we co-launched with Microsoft and Google Now when you go to um donate a project into the Pache software Foundation like um you know um open source is a really really interesting world and as folks like read into it and really understand like what

18:45what is open source and even look at um because a really key component of Open Source is governance um and and like governance from a community um perspective um for like safety even guarantees inside there right because there's actually been some interesting stories even just this year where the rug has been pulled out from

19:09underneath folks on on open source and a project can like disappear in a matter of 24 hours which which there would be like many in the community using and depending on oh it had a patchy um uh 2.0 licensing so I'm safe to use it you know Le but the community without governance or a project without proper governance can like vaporize and people

19:34don't don't recognize that so um we decided to donate because a key part to this right is having collaboration across the hoodie Community the Delta Community and the iceberg Community that's the only way a project like a one table xtable could survive or or win right and so we needed this project to be arms length from everybody be in a neutral zone where it's not like

19:57one of these commun communities could dominate or or have control over over the thing so we donate to the pachy software foundation and during that process um there are a lot of reviews that happen there's legal reviews um um there's trademark reviews um there's there's a lot of things that go on and one of these while there wasn't something related to trademarks

20:20or otherwise there's even a principle in the Pache software Foundation that like you should um also um be sensitive to other types of things so for example one table the name was very similar to the name of one house our company and one table is also very similar to the name of one Lake a new product that was launched from Microsoft and so because the two of us were two of the three that

20:43were like on that initial um list donating um um the project there was a recommendation from someone in the community of hey let's let's be careful and not name it something that's too close to these initial um committers here um give it something more neutral and so that's where we came up with a Apache X table um and now it's uh in the

21:05incubator program in the Pache software foundation so um I think you asked me two questions though so what was the uh second one U you asked about the name origin and the name and I think like how it's working like what's the magic behind this how it works yeah yeah okay how it works in the magic behind the scene so um the first thing that people think when they when they hear about a Pache X table is the thing like oh is it

21:28just another table format and there's that famous that popular um XKCD meme about like a standers or something and throw one more in the ring right um so when uh it's easy to think that way on the Forefront but actually when you look at what it is it's not a table format on its own at all like it cannot you cannot write data as an xtable you cannot read

21:51data as an xtable it it just doesn't exist xtable is only in the middle um and provides interfaces to um unlock the translation of metadata across these these table formats so um to explain how it works like we talked about un commmon ground across hoodi Delta and Iceberg it's all paret files in the hood and then a specialized metadata wrapper on

22:14top now Apache X table knows how to read the metadata layers of all three um table formats and Apache X table knows how to convert one metadata um um um setup to another um and so in in the operation of translation it doesn't have to read any parquet files it doesn't have to rewrite any parquet files so no data is

22:37duplicated um and also when you look at the overall storage sizes or performance considerations here right like in a in a 100 terabyte table your metadata is still only going to be a few uh megabytes like it's it's tiny in comparison or you know maybe it's tens or hundreds of megabytes inside there but it's tiny in comparison to the actual data and table size So reading

23:01that metadata is a lightweight operation Translating that metadata is a lightweight operation and so all of this can happen asynchronously um xtable in its first iterations here the way that you can execute it is you can um just package up a jar file and run that execution however you'd like to see fit so I've seen people in the community

23:23that will run it in line with the right operation and say every time I make a commit do a metadata translation or I've seen some people that'll take it asynchronously and Trigger it once an hour or once every 5 minutes or I've seen people now even AWS just published a Blog where they operationalize the inside of the um airflow offering that

23:45they have mwaa they're long acronyms that they like to use um so you can put it in like airflow and run it in trigger in a pipeline cycle even folks like Microsoft and snowflake are now using the project to together um they just announced at the snowflake Summit um that collaboration with Microsoft on one Lake and so even snowflake now when they write tables external Iceberg tables to

24:09one Lake behind the scenes Microsoft Auto translates that to Delta and vice versa when fabric writes data into Delta on one Lake it auto translates back to Iceberg for snowflake so they're using also the open source project um for for all their customers as well um hopefully that cribes yeah and I think it's amazing because like you don't have any

24:31dependency when you read or write the data and you only have to you build like have some some kind of maintenance on the side and doing the conversion in the background um which I think is super good for adaption because you don't have now to fight with every single vendor and you know every single instance to add X table on top of everything else that's right yeah it needs no it needs no vendor support to make the

24:55translation happen um it's completely independent a vendor could choose to use it like we use it in our product and now I mentioned how like Microsoft and snowflake now use it in their product so vendors can choose to like make more wrappers around but it's not necessary at all and and you can use it with any tool of your choice and and and I'm curious so you mentioned for example for hoodi um that

25:18you have some specific options to do streamings and low latencies um I would imagine we have the same kind of you know specialities for each for like iceberg is going to be good at doing something Delta is going to be good at doing something if you start using xtable do you have to kind of standardize on um let's say what's working the

25:40less well like if you know if if you have hoodie and you want to go with low latency appens and stuff like that then if you turn on xtable you lose all the benefits because now you have to go as slow as uh for example ice back on Delta on this kind of operation sure sure great question yeah um and and coming to like a least common denominator kind of factor right um I think that's a valid concern so

26:03also one one other um description of how it operates as well as in this flow you are choosing um basically like a primary writer and you're telling what um the other two like which other ones you'd like as secondaries to be able to read okay so you still um make a choice of do I want to write first in iceberg write

26:25first in Delta write first in hoodie um and then I can read it as all three um on the top right and um by so doing then on the writer side now you can um pick and choose like which of the Specialties you need inside those projects for right operations and on the read side get some of those other benefits because now even when you think of that that core based

26:47table and even how you run maintenance on that table because all these table formats need things like um like a cleaning operations or what Delta calls like optimize command or clustering operations compaction these kind of characteristics all the table formats need so the maintenance is run you you um the best practi is to run that maintenance in the format of how it's

27:11being written as well and so what what this comes down to is um um you can actually still benefit from some of the specialty edges that are Niche to the different projects that the other projects don't have even with X table involved so if you take for example um

27:33um um like if cuz here on this base table let's say we cluster that table and let's say one project has like a specialized fancy way to Cluster or sort the table um something like zorder otherwise maybe there's something else fancy um now you translate that to Iceberg the only thing that changes is

27:55the metadata layer on top all that data that has been sorted in the fancy way lives and exists and persists so when you query it your query engine can then still benefit from the way that the storage has been laid out underneath the covers um the edges where it exists today because there are uh limitations um to xtable and those current limitations do include um uh it not uh

28:20translating across merge on read uh base tables uh so that includes deletion vectors from Delta um and uh hoodie and Iceberg more call it more traditionally merge on Reed um um that way so those type of tables um it doesn't work quite yet that's something that's being worked on in the community um today but then otherwise things like um um partition

28:44Evolution scheme Evolution all these kind of things they can translate across across the projects okay so so just to summarize with the Apache ex table you can write any format translated to other uh to other format so no matter what you're choosing on Delta Lake Iceberg hoodie uh

29:06as a primary format you can simly have this interop interoperability across all of those formats but the main question is how is this different from uh Delta uniform yeah good question yeah so the um principles are the same um exact same principles and goals and things like that um and actually even a lot of the

29:28implementation is actually the same as well so cuz I I even um personally meet with and collaborate with the folks that are building uniform um over there and um because we're we're aligned to the same vision and goal and and and things and so we're we're open to share code we're open to share ideas and we're open to help um ensure that the

29:51interoperability on our different legs works works uh uh together so you wouldn't you wouldn't choose to use both of these projects together but we want we both want the three legs of Hoodie Delta Iceberg we want all three of those legs working really well and so we also share um across these two communities of uniform and xtable we share hey hey we solved this in a

30:14certain way and it made this kind of um um uh way to translate more sustainable um and and so we share back and forth that way so um because you know from from The Roots right um and some roots of where one table comes from some roots of where uniform comes from uh we've already been through a table format um

30:36Wars as as you might call them on um of our own and and now it's like let's move past this and and let's just get interoperability um and so that's why there's a lot of uh good collaboration going on these projects too all right yeah and I saw I think I saw there is a like a partar between uh uniform and and ex stable is that is that correct like a partnership

30:58um oh um yeah so yeah we collaborate we share code behind the scenes we share ideas we share development um behind the scenes and even Databricks um um has committed to help uh be on board and and support xtable the same ways and in reverse that xtable can support uniform so cool okay I just realized I did like a a French English with partan

31:23partnership it's okay I think I was follow I think I have one more question like if uh how can folks get started with Apache xtable and if you have any trainings uh L documentation links or maybe recordings that would be helpful if you can point the community where they can find those resources absolutely

31:46absolutely so um we have a good website um xt. apache.org let me just make sure I got that URL right if it's not the other way around yeah xt. apache.org and from there you can navigate to um documentation up on the header the nav bar on that website there's documentation there's a quick start inside there there's a Docker demo um

32:08those would be good places to start and try a little hell hello world example for um xtable conversion um uh then it also links to the GitHub page so to get started you go to GitHub uh clone the project um just get started right away on your own and then when it comes to um helpful resources Beyond document ation there's also blogs on the xtable site so

32:31you can go to blogs and those link to resources um maybe a great uh video or material is um you can check out even on the date in AI Summit I gave a talk there um about a pachy x table um and uh so that should be a great resource as well I gave a great uh demo and as you mentioned I think at the beginning of this interview as well um I had a demo

32:54that integrated with unity catalog um that was fresh off the press open source it became open source that morning um I put a little coding in on the side got it ready for the presentation just a few hours uh later um and so yeah you can check out xtable um a quick start demo how to how to get started that way and integrate with unity catalog amazing

33:17when did you co when did you implement this because I were with I was with you at the lunch then we took a cup together so so that's actually interesting story right so I wake up I was very excited to get to the keynote cuz I knew some like good things were coming and I woke up and I was not uh uh feeling great so I was at home um in I

33:39was in my hotel room um and I was watching virtually unfortunately and I was like bored on the on the side and I'm like okay it's open source let me just try it and so I I I opened the gith it was really easy to use actually and so seeing how simple it was to get started I'm like okay maybe I could try this maybe I could actually squeeze it into my demo um so yeah it only took an

34:02hour or two and the only thing that I was nervous about was actually like delivering it and the demo Gods being like happy with me that I'm throwing a last minute wrench into the equation it all worked out yeah yeah yeah it all worked out though and it still landed a good demo for the show so all right it's amazing well thanks a

34:24lot Kyle uh everybody try you know xtable go ahead write the data once in the format of their choice and you know you make you can make it available on every other format so go ahead and give it a try um and yeah Kyle thank you very much was super clear Y and we'll you know thanks for having me yep sounds good all right byebye good chat let's do

34:47it see y'all