Delta Live Tables 2026: Multi-Catalog, Kafka Sinks & Cost Optimization

0:04Right. Hi folks. So today we are with Bilal. Thanks for being with us Bil again. I think it's your second time. Absolutely. And we'll be talking about uh DT even if you have a nice DBT shirt which is super confusing. They're both threeletter words. So you know why why is it confusing? All right. So I know I know you have been working at Databricks on a lot of the data engineering parts a bunch of workflows

0:28and now focusing on all the DT integration and also I know we had like many many new release and new updates on DT so maybe we can start with that uh what's going on with DT what's the latest like what's the latest update yeah I think before I think before I have a an important question like we are calling them uh data life tables DT and pipelines so what's the official

0:51Yeah. So, uh, okay, I'll tell you the official name in a moment, but first I'll I'll tell you what the product is. So, that's actually the most important part. So, uh, you know, we call them DT. Uh, nice three letters, not you know, Delta live tables, uh, just DT. And, uh, for those of your, uh, viewers and listeners who don't know, uh, DT is a declarative framework for doing ETL. Now the declarative piece means that instead

1:14of worrying about orchestration, instead of worrying about infrastructure, you know, all those things, you really just tell it what to do, which is really the queries. These are the transformations you want to do in batch and streaming and then the framework takes care of the rest. So in some sense, it is in fact a little similar to DBT, but it's built on Spark. Uh and uh that's that's DT. Now we have been really busy over the last

1:37year and uh there's sort of two reasons for that. one is uh you know the business has grown a lot so the product has grown a lot it's got a lot of adoption and also we've been busy uh really polishing the product and this has for me the last year with the LT has been the year of polish and that's good news for customers because uh often what we hear a lot from uh our customers is actually this is a really really good

1:59product uh but I want you to make it more stable I want you to make it more cost-effective I want a better developer experience I want better errors I want better documentation that's really what we've been doing. So, uh, we've been building a more delightful product. Uh, and that's just, you know, uh, pretty exciting. So, I'm happy to get into that with you. And, and Bill, I know one of the common question is, hey, should I use DBT? Should I use DLT? What's the

2:22difference between them? Can you can you maybe share your, you know, your point of view like what should I be using? Yeah. Yeah. So, so first of all, I'm a big fan of DBT. I've been using DBT for a long time and, uh, they're wonderful. Uh, first of all, it's a great community. It's an open source community. Uh, we love that. So my answer is very simple. I think DBT is a great product. Uh whether it's open source or DBT cloud. Actually, if you

2:44haven't used DBT cloud, you really should check it out. And if you're using it, you should keep using it because it works really well and it works extra well on Databricks. Uh so in fact, DBT is growing fastest I believe on Databricks than anywhere else. So it's a great product whether it's the cloud version or the open source version. So keep using it. It's awesome. Um now in fact a lot of people don't know this but actually DT and DBT work together. So uh

3:08for the DBT users among you, you can use materialized views and streaming tables which are two of the tables and views that are produced by DT as incrementalization strategies in DBT. Um so that's super awesome because you get the benefits of uh DT the incremental ETL inside your DBT projects. Um now if you're starting off on DT on Databricks and you don't know what you know what to

3:31use, we would love for you to try DT. We think it's awesome. I think it's uh native to Databricks. Um, one of the superpowers of DT is it the way it deals with streaming and that's something DBT actually doesn't do, right? So, DBT is sort of agnostic to the execution mode. It's really was built for batch. If you have anything that resembles streaming uh whether that's incremental ingestion or incremental ETL, DT is a great

3:53choice. Um, so it's really we support both uh and both work super well on the platform. Bel you mentioned that Databricks spend a lot of time like polishing the uh the DT. So can you tell us maybe one of the top features that has been maybe announced or deployed during the last couple months? Yeah. Uh so actually what's what's really interesting my favorite uh feature is

4:18actually better errors. And that might sound strange because you know hey you don't think errors when you think features but actually every developer including me the thing that we spend time most on is actually not developing our pipelines it's actually debugging them. It's actually super common. So my favorite feature is actually better errors. And uh the way we did that was we actually are going error category by error error category and we're assigning

4:40codes to them. We're assigning human readable messages. We're integrating with the assistant. So when the error happens, you'll be able to diagnose the error. So that's that's not a sort of a traditional feature, but I think it's really important for the inner loop of the developer. Um I would say the second closest if I had to choose would be that we spent a lot of time on um you know when you run a DT we actually do this super cool end

5:05toend analysis of the graph. So the the directive the directed as graph with all the data sets and how data flows between them. We do type checkins many more things and what's really cool is that that thing is really valuable but it was slow. So we've made it like 90% faster in the last like 3 months. And we're on this mission to just make it as fast as possible. And what it means for developers, like I'm an engineer and the

5:28way I think is, hey, you're saving me time. I'm sitting in front of my computer. I'm developing this DLT. It used to take 30 seconds. Now it's just, you know, going to take a couple of seconds. That's pretty awesome. So that's the kind of stuff that, you know, uh, costs wall clock time. But to me, those are pretty awesome features. Yeah. And I think uh like being able to check all the SQL of the Python syntax and make sure everything is working and making sure it's super fast like it's so

5:50nice because you would be changing something just hit enter it's going to recompile everything you know you're safe and you go you can go to the next one where I know it used to be a little bit slow but now it's making everything like way like way more interactive in the way you debug stuff. So that plus the error that's really and I think also maybe I think you missed like I for me my top my top feature was the what we call direct publish mode like the

6:13possibility to write to multiple catalog. Yeah. Yeah. This is this is actually uh super interesting. So actually we're calling it um I don't even that was just an internal code name. So here here's the thing. So uh now your DT a single DT can write to multiple cataloges and schemas. It sounds like a really small thing, but this was sort of surgery in the spark code. Uh, which is pretty

6:36awesome. This is another one of those, you know, uh, from a from a feature perspective, it's it's you just use it, you just write to different catalogs and schemas, but the implementation was very complex and kind of a long road. That's also going generally available pretty soon. That's pretty exciting. And I think one of the main questions that people were asking like, okay, when I'm using a delta DT, it creates behind scene what we call streaming tables. So those are

6:59like tables that are append only delta enhanced to handle streaming part which means any for example SQL and like data analyst can use streaming tables to do streaming and then we have materialized fuse so like uh they are do like they are better handled to better designed to handle CDC or like incremental changes but those type of tables can only be

7:22let's say used uh inside Databricks. So how can I expose those kind of tables outside Databricks? Yeah, that's that's a that's a really good one. This actually uh something that we hear a lot. So as the product got used more and more um so uh we're fix we're fixing it actually. Uh so we have a capability called compatibility mode. It's in private preview right now and that lets

7:44any external reader anybody who can read a delta table actually read these uh materialized views and streaming tables. And uh this is in preview. You should contact your account team if you want it. working hard on getting this out the door becoming generally available and uh the roadmap for this capability is pretty ambitious because uh we currently don't support deletion vectors but we are working on that support. So uh anyway lots of work happening here

8:08and with that you know you build your DT and Databricks and then just add a couple of table properties and then you know voila it's like ready to go and you can read this from anywhere. Uh it's pretty pretty cool and I think one of the thing I heard also like as kind of a complaint on the product was that if you publish things on DT you would have issues to you know apply things like role level security and things like

8:30that. Is it is it like is it now fully working? What's the latest on that? Yeah. No that's that's a really good one. Um so yeah so uh for all intents and purposes streaming tables and materialized views are just like regular tables and views and Databricks now. So you can apply role level uh role level security column masking you can do liquid clustering you can get a change

8:52data feed out of streaming tables soon you'll be able to get a change data feed out of materialized views that's on our road map as well. So yeah so we've we've been on this mission to make them fully look and feel just like tables and views. So that's already available now. Uh so you can do liquid clustering now on your uh streaming tables if you wanted to and MVs. I think I haven't tried yet this feature. I I have to do it. And and what about the cost? Because

9:16I've seen uh if I'm new to Databricks and I go and try to create a DT pipeline, I see that the um the DBUS are a little bit higher. Is it because they are doing some optimizations on your behalf to make sure that the cost remain remain like uh low like what? Yeah. So, so there are a couple of things that are happening and I think um you know, so DT does automatic

9:39incremental ETL. So what that means is that uh we only we do our best effort to only process the new data. Now if your data set is really really small you can just sort of recomputee everything historically. It doesn't matter right if you have you know let's say you know gigabyte of data it doesn't really matter you can just throw it all away and recomputee it but for large tables this can become extremely expensive. So DT has this engine which is super cool.

10:02It's called Enzyme and Enzyme actually analyzes the query plans and it actually figures out now it actually figures out like what data has changed and how to do the ETL incrementally. Now it doesn't work for every single workload, every single query, but we've spent a lot of time increasing the coverage of Enzyme. So we're super proud of that. Um now

10:24still you know we want to give our customers the most price performant offering. So, we are doing a bunch of things to make the total cost of ownership of DT go down and just be more affordable. Uh, we want every customer to use it. Uh, we're doing a few things here. Uh, I'll give you sort of a super high level. Uh, on the high level, in the next month, you should expect the total cost of ownership of a DT to go

10:46down. Uh, you should use serverless. All of these enhancements are coming out only in our serverless offering. Uh, that's how they're implemented in the serverless fleet. Um so all the all the enzyme engine enhancements are there. Uh the way we build for the actual um photon as well in DT is in serverless. We're and releasing enhancements there

11:08and uh we will also be releasing a new serverless mode which will let you trade off uh long latency for compute with uh lower cost. That's all coming. So think of this as a package of capabilities that are all aimed at reducing your cost. And this is all happening in the next you know few weeks. So uh it's pretty good news for our customers. And again you can also try to try to monitor

11:30the cost using system tables and the different dashboards you do have there. So you have nothing to worry about this. And and there's a new pipelines system table. So that's actually pretty cool. I believe that's in preview. Uh and that wasn't there before. So it was kind of really difficult to figure out the cost. Uh this is there's a whole bunch of other metadata in there as well. Yeah it was part actually of our the last discussion we had together in Slack.

11:53And and I think I have uh another question like like datab bricks announced lakeflow connect and or like lakeflow uh lakeflow product and it's built on the top of uh DT. So can you give us maybe more insights about lakeflow or the three components first? Absolutely. Yeah. So, so LakeFlow is uh

12:15the uh unified, streamlined and performant uh data engineering uh product from Databricks. Uh it has three components. The first component is LakeFlow connect and lakeflow connect is no code pointand-click injection from enterprise applications like salesforce, workday, netswuite. So both the structured data sources as well as unstructured data sources like sharepoint and uh such such things. So

12:40the whole premise of layflow connect is you you know point and click at the source and let us take care of ingestion. Um and that by the way that includes databases like SQL server and more to follow. Um now once you have that data in Databricks the very next thing you want to do is to transform it you know filter clean aggregate it and that is actually uh DT that's the second

13:02piece of it right that's the second the declarative ETL and finally to get this all into production now you have to orchestrate this you have to run this on a schedule maybe you want to run this on a trigger you want to combine this with other let's say nonel things like you know you have an analytics notebook or a query maybe you want to train a model That's actually LakeFlow jobs and LakeFlow jobs is going to uh ensure that

13:24everything is up and running. It's scheduled. Um yeah. Oh and then sorry your question was hey why why does connect build on top of DT? It took me a second to remember your question. Yeah so actually it's a really really uh simple answer because ingestion data ingestion is almost always incremental. Uh you almost never want to it's only in cases where there is an operational failure that you

13:47may want to reload historical data. Uh so when your injection is incremental that's exactly what streaming tables are. Streaming tables are designed for incremental inserts and appends uh sorry appends into uh a delta table and then because we're using we actually use streaming tables uh under the hood for lakeflow connect you can actually do incremental ETL off of them. So the increment the ETL when you when let's

14:10say you know you have a let's say you have a Salesforce orders table that is arriving incrementally the you can then have a streaming table or materialized view in a DT that incrementally processes only the change data. Um this for many workloads removes the need for manual incrementalization which can be very costly and difficult to reason about and actually costly if you get it

14:32wrong. So that's why we decided to build on top of streaming tables and materialized views because they're the right primitives for ET. And um I I'm going to bring you back again to DT, but can we use the DT for um low for like um for streaming use cases with a low latency? Yes. Oh yeah, that's that's a really good one. Um so

14:56DT will soon support a number of expanded semantics for streaming. Um, so specifically we're going to support something called for each batch. So if you've done structured streaming that's going to be supported. Now if you want to go really fast, if you want to go, you know, down to let's say 50 milliseconds of latency or 100 milliseconds of latency, you have to stay in structured streaming because in

15:18structured streaming we're adding something called real-time mode. And that real time mode uh is going to support you know uh the predictable low latency. Um this is all part of the lakeflow umbrella. So you actually get a lot of the benefits of Lake Flow. Um bringing that mode to DT will take us a little bit longer. That is something we want to do. Um

15:41yeah, I think you covered all the questions I had on top of my mind. What about you Q? May maybe you would last one on the like on the input and outputs. Uh I I know I know folks want are curious to know about you know um sync and flow. Yes. Um so maybe can you share a bit on that and how it's working and what's new there? Yeah. Um so um let's see uh so the general idea here

16:04from a you know for very simply uh we're just expanding the ability of uh DT to uh write data to multiple different places. So previously DT could only write to you know streaming tables. A very common request we get is hey I'm reading from Kafka and I want to write to Kafka. Can I do that? uh or a common one another one is hey I'm reading from

16:27you know uh Kafka and I want to write this to a delta table like can I do that and you couldn't do it so uh we've added these uh these uh new types of um this API called managed syncs and with those LT is now capable to write to not just streaming tables it can write to delta tables it can write to Kafka so if you have a that's actually a really common

16:49use case so that's pretty exciting to for that to be supported in DT awesome thank Thank you so much, Bal. Thank you for and looking forward to having you for the maybe third time after the data summit. My pleasure. Thank you guys. Thank you.