Future of Data Engineering: Declarative Streaming + AI

0:00instead of coding in python or Scala you're going to be coding in English you try it you know it's not working you change the query try again the test works and like it's super fast so you'll say I want to copy data from there to this spot in the catalog and the system will take care of it for you we're simplifying more and more this user experience there is no migration there's no breaking changes you'll get incremental improvements to the existing

0:22products as you're using them today hi folks hi Q uh today we're hosting Michael armur at distinguish engineer at Databricks he designed spark SQL then streaming then Delta and now he's focusing on the next generation of data engineering and my question is

0:46is there anything Michael doesn't do you're too [Laughter] kind all right so um maybe to get it started I wanted to Michael to talk about the like the overall data engineering ecosystem like Spar is um I think it's 10 years old maybe a little bit more little more I think yeah yeah I

1:09was wondering like what do you see in the market like uh nowadays if you are a new data engineer which tool would you be using like spark but do you do also python do you do do you do Scala I don't think so anymore really maybe sequel to use AI to build everything I don't know what what how do you see the market yeah I mean well I think if you look at even just the history of spark we have been slowly

1:33upleveling spark over time so you know like before spark there was Hadoop and you had to spell out every little detail you had to have these different like output writers and input wrer readers that would control exactly how the data was serialized you had to write out your algorithm in you know full detail and Spark made that a lot easier it gave you these like simple functional commands uh where you could map and reduce and do

1:55this stuff but all in just like a few lines of scholar code um and then we added spark SQL to it and like that was a big simplification where now you could use data frames in Python data frames in Scala or just plain SQL commands to run it and so I think this this trend will continue to have kind of higher level abstractions on top of it and I think you know really where we're probably

2:17heading is instead of coding in python or Scala you're going to be coding in English you will talk to an llm in English which is this like messy uh ambiguous language and the llm will produce python Scala SQL one of these unambiguous languages um and that will be that'll kind of be how you're working and I think I think that's good that that Trend upward is going to continue

2:39and what what's the big chunk of um like language you see running on the bricks is it SQL or is it python um I mean you know I think I think that it's pretty split but definitely SQL and python are the ones that are growing and schola use is declining over time and I think it really between SQL and pyth Pyon it's more of a Persona

3:01thing it's really if you know SQL and that's your preferred tool of choice you like to you continue coding in Sequel if you want to do more procedural coding if you have like a bunch of steps you want to orchestrate them together and you want to modularize your code uh and you want to interact with libraries like pytorch or these other things then then you're probably using

3:23Python and DLT uh was introduced two years ago and can you share a bit on the story around DLT so you know yeah DT has come gone through quite a few Evolutions over time but really the goal was exactly what I said at the beginning we want to keep upleveling the task of doing data engineering so you know when we were building spark SQL we were building this

3:46very generic tool for executing declarative SQL queries and the magic trick of that that declarative language is instead of saying like in order to compute an average I will first do a sum and I will do a count and then I will divide them you just say give me the average and the system figures out the right way to do it where the shuffles need to go uh where it's going to do partial aggregation all of those different steps just kind of happen

4:08automatically DT is the next level where we want to make the entire process of doing data engineering declarative so now there are actually verbs for create this table and create it up to date uh and keep it up to date or do change data capture and handle all of the outof order data and keeping history and all the other things you need to do you just say what you want to have happen

4:30and the system will figure out how to do it um it started off as you know if when we first created DLT the product name was probably going to be serverless streaming the idea was just take spark streaming and make it easier to run add all of the retries and the metrics and the auto scaling and all the things you need to do to run a a streaming application without having to tune a whole bunch of knobs but what we quickly

4:53Learned was continuous streaming is a tiny fraction of the data engineering people want to do it's great when you want low latency but it's pretty expensive um and also streaming has this this fundamental it's both a limitation and a superpower streaming promises it's only going to read the input data once and so the types of things you can compute with it um are limited but also it's

5:15guaranteed to have a certain performance and so over time DLT evolved from this kind of very focused on streaming pipelines to a generic tool for doing data engineering but declaratively yeah I feel like it's a like we say streaming but it's more incremental Pipeline and then you can decide uh like the latency do you want it running every day or like every

5:37second it's up to you but like the key is an incremental incremental pipeline that's exactly right yeah that's one of our magic tricks when people hear the word streaming they often think Super complicated super expensive in order to get the lowest latency and Spark can do that and DT can do that especially with the Advent of real time mode which is something that's in private preview right now where we actually now able to

6:00run continuous streaming pipelines with subse latencies which is you know competitive with some of the best streaming platforms that are out there today um but the really cool thing about spark and the thing that nobody else kind of in the streaming space can do is you can tune that cost latency trade-off to me streaming does not mean necessarily low latency it can mean that but it's exactly what you said it means incremental computation it means reading

6:23the input only once and it turns out a lot of data engineering is exactly that when you're ingesting data you you only want to read it once you know when you're doing Transformations you only want to read the input once and with triggered streaming inside of uh you know DLT we could do that super easily and the best part is unlike when you were using kind of plain spark when you go back between triggered streaming and

6:45continuous streaming you don't need to make any code changes so if your business requirements uh change over time it's one click in the UI to change it to a a continuous streaming Pipeline and get those super low latencies so so doing incremental like if it's new data and you just want to open the data to an existing table like I can see how it's kind of simple you just consume the new data and you open

7:07that to the table but in most of the pipeline we have uh like you want to do something else you want to do aggregation maybe you want to join with other tables creating views and stuff like that and I think that's a really hard stuff because most of the time you if you do it yourself you would have to recompute everything and I know DT has some magic behind the scene to you know to help with that can you can you share and tell us how it's working yeah

7:30absolutely and this is I think one of the biggest challenges people have when they're understanding DLT is like DT has several different abstractions it has tables that have streams writing to them uh and it has materialized views and really the tradeoff between these two abstractions is like I said before streaming has this limitation and this superpower that it only reads the input once and that limits the types of

7:53queries you can write so you can do aggregations but you can't really do aggregations where you need to keep arbit amounts of State in memory so for example if you wanted to do like histograms over arbitrarily large data sets streaming is not a super great thing for that you'd want to use like a sketch and do an approximate histogram instead you want to do uh time window averages where the data arrives in a bounded amount of time streaming is a

8:16perfect case for that would you want to do outer joints window functions these more complicated things then materialized views are the right solution materialized views are still incremental in many cases and as we improve the enzyme query Optimizer that's our that's kind of the new system we put on top of catalyst that not only understands how to optimize queries but knows how to update existing results to queries efficiently when the input changes uh but uh enzyme can run all

8:40types of queries it can do joins it can do outer joins it can do aggregations and the magic trick is it's allowed to read the input again if it needs to so let me give you an example of where that would be necessary if I'm Computing a Max and the maximum value is deleted from the input table well now I have to go back and look at the input table and find out what the next maximum value is

9:02that's something that streaming could never handle it can't do retractions it can't understand the idea of data disappearing because that would require sometimes arbitrary computation materialized views will automatically take care of that so in the cases where it's possible it will be as fastest streaming it'll do exactly what you said it'll read only the new data append it Downstream everything will be fast but in the cases where something changes or in the cases where there's an expensive

9:25uh operation that requires extra work materialized views will also do that trans parently so there's this tradeoff between guaranteed performance and expressive power and what that means is you can really kind of no matter what your requirements are you can get DLT to do them you just need to kind of think about what's more important to you okay and like DLT like in the past we could

9:48not choose any like what we call the run times you just need you just choose let's say the uh the number of workers and the type of machine and that's it now we have uh server list which also gives like simplify more the uh deployment of DG pipeline but can you tell us more how does DT work with serverless and how this can help the

10:10customers optimize their cost but also the performance yeah totally so serverless is you know not only a DLT feature it's something we're kind of doing to simplify the entire data brick platform and the nice thing about serverless is you no longer need a cloud account you no longer need to worry about IM roles or all of kind of the setup of you know whatever whatever Cloud you're running on we take care of that for you and it has a couple of

10:34magic powers so one is you can now sign up for data brecks with only a credit card and go all the way to processing massive amounts of data sets without managing any other infrastructure you don't need to worry about quotas or any of that kind of stuff um but also serverless has a couple of magic tricks it's got a super awesome autoscaling algorithm that works not only in DLT but across all of the other products and it has and the one of the kind of cool

10:56parts of serverless is we only charge you for the compute that you use so it's on us to figure out how to allocate those machines and how to bin pack them and do all these other things and you're only ever paying for the capacity that's actually doing useful work for you um serverless also has a bunch of performance features since we manage the entire stack we can do things uh more quickly in some cases so especially with

11:19streaming workloads we can pipeline different uh we can do streaming pipelining and execute multiple micro batches in parallel we can automatically tune the admission control parameters things like Max files per trigger since we know exactly how big the cluster is and how much extra how much extra space there is and how much space there is in the warm pool we kind of we take care of all of that automatically and so in some cases we've seen that serverless can be

11:43both faster and cheaper for large scale SK streaming ingestion jobs and simpler so so we have we have the LT and webs you can use DT like as a as a product and create your DT pipeline but then you can also create create materialized view which I think are using DT kind of in the backand uh I know also when you create Victor search

12:06for example we using DT in the back end how we are we starting build to build everything on top of DT like as an engine and then offering higher level um function to customers is that how it's you know working yeah so I think the right way to think about this is you know DLT is like it's it's a brand name and it's an enabling technology and what DLT that that engine that kind of that runtime is good for is it's good for

12:29doing data engineering it's good for reading data from one place transforming it and writing it to another place and it simplifies a lot of that work and of course even some of the products that we do internally have that exact same problem so uh system tables are built with DLT we do our own internal security with d LT when we're moving data into an online table or into a vector a vector

12:52index DLT does that movement when you're ingesting data using um lak flow connect um that also uses DT um but really the abstraction when you when you know you as a customer are using DT there kind of a few things you need to know about you need to know about streaming tables and materialized views which we talked about a little bit uh you need to know if you

13:14want to do Advanced things you need to know about flows which are actually like it's it's streaming but it's also things that aren't streaming like for example doing onetime backfills so if I want to have a single table I want to do a one-time backfill and then I want to read from two different kof in different regions and WR into that table then you're going to start using flows and then the final DT abstraction is a pipeline so streaming tables materialize

13:36views you can use just on their own you can just go into DB SQL say create streaming table say create materialized view and the system will automatically manage everything you don't need to know any other Concepts just one one line of SQL can build a highly scalable ingestion pipeline that's automatically managed but when you're doing like a large data engineering task you want to do the kind of full software development life cycle you want have source code

14:00that defines all of the tables you want to Version Control it you want to deploy all of those assets together and that's where pipelines come into play a pipeline is just a collection of streaming tables materialized views and flows that are deployed and executed together and so just think of it as like a a management abstraction on it so kind of just to summarize DT is this enabling engine that lives under the covers the

14:22important things for you to understand are these streaming tables materialized views and pipelines yeah and since you mention um like mvs and STS um when when it comes to developing code I know it can be it was it was in the past a bit tricky to debug the code when When developing on DLT and I've seen that we are

14:44bringing we're simplifying more and more this user experience but can you tell us more on this uh on this part yeah totally so I think one of the first steps we did was these DB SQL materialized views so now you don't need to do any extra setup you just run a single SQL command it runs like normal and it creates a DT object but even for more complicated pipelines we're

15:06spending a lot of time on the developer experience this quarter uh we recently moved we recently decided that basically this is one of the we really like this underlying technology but we think the user experience of doing development has been holding us back a little bit so we recently did a pretty big reallocation of Engineers to get a lot more people working on improving that developer experience and there's a couple of things we're working on so one is DLT

15:30has this magic power called analysis because DLT is declarative we can actually analyze all of your code without actually running it so we can look and say are all of the tables in this pipeline valid are all of the column names table names data types functions correct without actually doing anything to the production data and that's something that's really only

15:52possible when you have a fully declarative language you can't do that with something Turing complete um and so you can so basically what the lets us do is when you're what we want to do is when you're editing you can run these validate commands but we want to make that validate operation so fast that it can be continually running in the background so we have a bunch of benchmarks that we're doing right now we basically once a week we're reporting

16:15all the way up to Ali our CEO exactly how long it takes to analyze a single table how long it takes to analyze python versus SQL how long it takes to analyze a table a pipeline has 100 tables in it and we're going to spend a lot of time optimizing that so doing things in parallel skipping steps where nothing is changed uh removing redundant work that we're doing and we want to get that to be super fast and I think what

16:38you're going to end up with is something that looks a lot like a cloud ID for doing data engineering um and I'm pretty excited about that because I think it's it's I think we can actually kind of you know today people are very used to notebooks and there's this really nice interactive experience but also you have to make sure that when you run those notebook commands you're not breaking production and I think that's like that's a that's a trick that's like

17:00that's a a pretty dangerous thing and maybe not the best way to do it's a great way to do I think data science but it's not a great way to do proper software engineering and so I think with this new investment in the development experience in the Databricks workspace we might actually be able to Leap Frog notebooks in terms of the what it's like to do development of production data pipelines and and what about the unit test I think unit test has been like a

17:23challenge forever in data engineering because you have to bring the data you have to test the transformation the input the outputs yeah that's a great question you know and it's actually something that we spent a bunch of time prototyping when we built the first version of DT you know like the first version of DLT was this thing we like I built on my LTAP that just reached out and used datab bricks apis to like run a job um and that first version had this

17:47really cool unit testing framework because again another magic trick of declarative programming is we can substitute things in the graph so we can say take this data flow graph that you've defined this table reads from this table reads from you know wres to this table um and we can say well actually that table is the you know the production data let's substitute in this sample data instead and see what result comes out at the end so we don't know

18:09the exact shape of this but we do really want to build data tests where you can say given this input make sure this output exists and have tens or hundreds of those examples that the system runs automatically for you yeah that would be awesome and if you can have that and and I don't know kind of live trying it you try it you know it's not working you change the query try again the test works and like it's super fast I could

18:31see how it could speed up everything definitely and I have a tough question I think this one cames a lot uh is DLT going to be open source at some point yeah so you know data breaks has a long history of doing open source we've started around uh you know this open source project spark the first major

18:53project I worked on at data brick spark SQL I remember the day that we opened that like I think it was 11,000 of code PR against uh Apache spark to kind of Open Source that as well um Delta was open sourced um so you know we absolutely want to open source DT it's just a matter of engineering uh we need to make sure we have the right engineering capacity and right now our

19:15focus is on making sure the experience inside of Databricks is as good as possible but we really like to work with the Apache spark Community we think that this is a fundamentally better way to write data engineering programs and we' love to make those kind of the standard apis and Spark for doing it and have some kind of simple Runner and open source that could run DT programs as well um so stay tuned it's something we

19:39absolutely want to do but right now our focus is on making sure the experience in data breaks is as good as possible the one other thing I'll say is the most important parts of DT are already open source the data that DLT produces is stored in Delta and is available in unity catalog that's those are both already open source the Transformations you're writing the nice thing about

20:01spark is it's not the nice thing about DT is it's not actually a new language for writing Transformations it's just spark SQL it's just data frames and to me the data and the Transformations that have your business logic that's the most valuable part that's the part that would be very expensive to port to another system uh but we want to make the whole all of it all of it open source including the runner it's on

20:23white and I see also often customers like asking question about DLT uh and and DBT uh but I feel like DBT is just using another Hood the DT computation at the end uh but maybe can you can you explain us how it's working like and and as a user like do you have any Lo in starting to use DT or DBT and how how you should be thinking about it yeah so

20:45I love DBT you know I think it's a pretty I think we have very similar goals between the kind of the DT project and the DBT project this idea that rather than managing your data infrastructure as a bunch of ad hoc commands you you run in production that's just not the right way to do it you want to have a Version Control description of all of the tables that are going to be inside of your catalog

21:07and you know DBT and DT are kind of very similar in that way um the key difference is you know DBT runs outside of the database engine so it basically you know it does like ginger templating to build SQL queries that it then executes against some other engine and that limits what they can do you can't it's difficult to incrementalization or things like that um but fortunately

21:31you no longer have to choose uh there's a Databricks connector for DBT or the datab brick adapter um and it creates two new materialization types streaming table and materialized view and so if you are already familiar with DBT you should absolutely keep using it um all you need to do is use those materialization types and you get can get all of the power that is only

21:53possible when you actually own the entire compute engine um and if you're not using DVT you know I do think DT kind of solves a lot of the same problems it makes it possible to kind of declaratively specify an entire graph of dependencies and automatically keep them up to date that's perfect and I think now we

22:16will talk about Lake flow which also is going to be built on the top of DLT if I'm not wrong so can you give us an overview on what lake flow is yeah so you know Lake flow is going to be the next evolution of the data engineering products inside of Databricks it always used to be that you know there was Databricks jobs or data

22:40bricks workflows there was DT and they were seen as these kind of very separate entities and um you know part of that is one of them is they're just very different ways of writing a notebook to do data engineering versus writing DT is kind of a different way of thinking but really I think that that that dichotomy is wrong um and the idea of Lake flow is we want to evolve this so that it makes it very easy to use the right

23:04tool for the right job and so the way I look at it is there's Databricks jobs uh and which are you know now going to be called Lake flow jobs so really DLT or lak flow is the evolution of all of these Technologies it's not just DT it's workflows DT uh Lake flow connect all of these things kind of wrapped into one coherent product and so the way I like to think about it is there's you know Lake flow jobs and you should always be

23:27using that for production a job is the way that you run something reliably either on a schedule or when some other action happens like a table gets updated a file gets uploaded some other job finishes you should always be using Databricks uh Lake flow jobs for that then there's notebooks notebooks are great for uh doing data science and investigation you should continue to use

23:50them that way notebooks are also a great way to run Legacy spark code if you've got data engineering that's already working and you don't want to change it it absolutely that's the right way to run spark programs and I use the word program here kind of very specifically when you have procedural logic that says if this then that uh you know o overwrite this partition where you're actually telling the system how that

24:12data engineering should be done you should continue to use notebooks and you should continue to schedule it with L flow jobs and then what we're what we're doing with L flow is we're now moving DLT kind of into this same UI so that you can seamlessly create materialized views streaming and pipelines all within that same experience so you know often your data engineering job might actually

24:34have all of these aspects if you want to start by downloading data from an FTP server well that sounds like procedural logic so you should use a notebook for it and then if you want to ingest that data into a table well that sounds a lot like a streaming table with autoloader you should use streaming tables for it and if you want to transform that data to build aggregations uh that are going to power a dashboard you should use a materialized view for that and a dashboard and you can put that whole

24:57thing into a lake flow job and and what about what about AI because I've been recently I've been using cursor AI uh like to do all my python stuff like it's so amazing like I'm just you know giving a few instruction like I got the python it's up and running even SQL uh are we going to inject a lot of AI within leg flow so that you can super eily build your pipelines and set up

25:19everything yeah absolutely so you I think the AI is already pretty good at writing SQL and it's pretty good at doing spark data frame operations and so those parts should work seamlessly already you know inside of Lake flow uh and whatnot um what we're working on is you know it turns out these large language models were trained you know on corpuses that were collected several

25:41years ago before the DT syntax existed and so we're working really closely with the Mosaic team and the data BRS assistant team to uh make sure that we train it on this new syntax that we have produced so they can provide good answers for that and that's actually kind of an effort that that just started this quarter to really come up with a regiment for measuring how good is it at understanding our specific syntaxes that we've added on

26:04top of spark um and how often is it giving good answers there and how can we improve that with more training I I can do you think like if you are a new data engineer do you still want to learn how to do python or do you think like in a few years it's only going to be you know write some English generate some you know Automation and that's it well it comes back to kind of what I said before you

26:27know English is this messy ambiguous language whereas Python and SQL have well- defined semantics you know exactly what they're going to do so I think it's never a problem to be able to read and understand that ambiguous thing I think that will continue to be a valuable skill because you'll be able to say is what the AI produced correct um but will

26:49you be writing a lot by hand I think that's going to be less and less I will say even in my own like stuff at work or the stuff I do in my free time I write less and less codee I let the AI write it I look at it I tell the AI how to correct it and then I then I use it um because there's just a lot of tedium in programming that goes away when the AI is writing it for

27:11you yeah it's super super interesting like and by the way things you mentioned about the data breaks assistant and DT this something we talked about recently with Weston is leading the data breaks assistant team and he told me about this U this new project to make sure that DT uh is well un understood by the assistant I was just curious to

27:35understand uh we did acquire Aron or Arion I'm always struggling to AR I asked the founder the authoritative pronunciation okay so for um how uh will this integration of archon works with Lake flow because I've seen like Arun used to have a lot of connectors and now we're deploying um I think now we have

28:00uh I think maybe service now will come next quarter we Google analytics Salesforce and more so how does work this integration totally so all of this will live under like the lake flow connect umbrella and just you know for people who haven't heard of it Lake flow connect is going to be a no code pointand click ingestion experience built into Data breaks so you'll say I

28:22want to copy data from there to this spot in the catalog and the system will take care of it for you it uses DLT under the covers but you don't need to know anything about DLT in order for this to happen um the archon acquisition was super valuable for a key piece of this which is getting the change log from these different databases so DT has always had this pretty cool technology called apply

28:45changes into where given a change feed that has a primary key and some ordering we will create a table that has the latest upto-date copy no matter what order that change feed arrives in even if parts of it are than others and a bunch of bunch of kind of complicated distributed systems programming stuff there um but what we didn't have the part that we always left up to our user was how do you get that change feed and

29:08that can be super difficult you have to pay for Golden Gate you have to set up open source projects like deum you have to figure out a way to like get them up into the cl to get the results up into the cloud with Kafka storing the data on S3 you have to build an ingestion pipeline to pull it in and what we're working on right now is taking that super awesome technology that we got from AR on and building it into the datab Bri platforms that we can handle

29:31that whole step you point us at the database we get the change feed we run apply changes and we create those tables for you and we'll be able to do this whether your database lives in the cloud that's in uh you know in private preview right now for SQL Server already or on Prem we'll run what we call the ingestion Gateway inside of your data center you know behind your firewall and it will upload only the data you want

29:54into a secure location and unity catalog where we will then apply app it into a Delta table um so archan is the thing Aran is doing is it's giving that that change feed uh you know for all of those all of those databases and then alongside of that we've been building connectors for a lot of SAS services so uh service now Salesforce Google analytics uh SharePoint Google Drive

30:16whatever it is we want to build a connector for it if we don't have it talk to your account team and let us know because we are we're rapidly building that out you know it took us a while to get the infrastructure in place um and now that we have a couple connectors out there I'm really excited to kind of start doing the like breath for search of the space uh really building all of those out and Michael one question so the inis pipeline that we have to do all the CDC and you know

30:39ingesting all the changes is it running uh with DT I mean under the Databricks yes yeah it's all running on top of DLT like I said you don't need to know anything about DLT in order for this to happen uh but the nice thing is if you are familiar with DLT and you want to do monitoring uh it's the same event log it's the same stuff so if you are a more sophisticated user that's already familiar with this part of it you can

31:02absolutely take advantage of that and have one single way to think about how is my data engineering proceeding what's data quality look like all that kind of stuff but if you're uh you know just a you know if you're newer to our platform just learning SQL or you're even just using the assistant lak flow connect is for you because you don't need to do any programming you don't need to know anything about pipelines or streaming

31:24tables or any of this stuff all you have to do is authenticate with the sources system and we'll take care of the rest and if I'm not wrong system tables is all is also using behind the scene DLT and then sharing the results with um using Delta sharing between our control plane and the uh customers workspace yeah that's right these are you know these are like implementation details

31:47but I do love that we eat our own dog food and use our own software in order to do it uh you know it turns out Databricks is actually one of the biggest customers of DLT um and yeah absolutely some of the system tables are produced with it and like you said yeah we use Delta sharing to share it securely with our customers um so always always fun to see technology Reus like that and you mentioned at the beginning that all our

32:10offering ser and that's that's that's true of course but is Lake flow going to be mainly serverless or not well you know I one of the key things of lak flow is we really want to evolve the existing products I know that like moving to Unity catalog and other stuff like that those are big migrations for customers migrations can be painful um and it's

32:33super important to us that with Lake flow there is no migration there's no breaking changes you'll get incremental improvements to the existing products as you're using them today um but a lot of the goodies a lot of the things that make Lake flow super easy to use will be things we can only do in serverless when we control the entire stack so um even if you're not on serverless uh you know

32:56you you will still get a lot of benefits from lak flow but in order to get the simplest best experience you should absolutely be looking at getting serverless I think it's really a game Cher in the things it allows us to control and the things that we can control are the problems we can take away from our users and so they don't have to worry about them anymore and they can focus on what you know uniquely

33:18they know about which is their domain and how to get value from their data that's that's right and I've seen I think a lot of in I think a lot of several features are coming next quarter if I'm not wrong we'll have the possibility to write to multiple schemas we will be able to get rid of the live keyword and yeah so you're you're

33:41talking about direct publishing mode which is the weird internal name for what I would call making DT simple so you know when we when we built DLT we spent a lot of time looking at dbd and and other projects like that and dbd had this way of ref referencing dependencies that were inside of the same graph so you would say like you know in DBT it's a ref uh where you say ref give me this

34:05table and in DLT you would say live dot give me this table and it would kind of find the right version of that table but what we realized is that was a we talked to a whole bunch of users that was confusing them it looked like you know they didn't want to learn some new thing they wanted to just use SQL um and this also came with a bunch of restrictions if if a pipeline is targeted at one

34:27specific schema well can only rate to that schema and so the thing that we are private previewing today is a dramatically simpler way of working with DT so it's basically it becomes just SQL we realize that we own the entire engine and so and we own the entire catalog so we don't need these extra affordances to tell us where to look for the table we can just look up the table

34:50in the catalog so at direct publishing mode so first of all no breaking changes your existing almost all DLT code will continue to work we'll just ignore that live keyword um and what the stuff that is coming is you'll be able to write SQL just like SQL and it will follow the exact same rules as if you were working in a notebook or in DB SQL so there a pipeline will have a default

35:12catalog and schema if you don't if you just say select from table those defaults will be used or if you say select from schema. table that default catalog will be used um and same thing for when you say create materialized view the default catalog and schema will be used um you could just reference tables if they're in the pipeline we'll detect the dependency if they're not in the pipeline we won't we'll just read it

35:36from the catalog like normal um and finally if you want to write to multiple schemas all you need to do is save what those schemas are so if you say create materialized view A.B.C it will create the materialized view in that location and you can use all of the metaprogramming and parameterization inside of DT to kind of control that and do cool things to have a separate development copy versus your production

35:59copy and all that kind of stuff yeah and I've seen also the cool stuff about the sync also I mean I've been following the road map for for a very long time no these are good questions you know and the sync is part of a more General effort our goal is to have full parody in what you can do in structured streaming in DT you know like

36:22I think notebooks are great for doing kind of like I said procedural data engineering but streaming has all always been pretty declarative the only part that is not declarative in streaming in a notebook is calling do start and you controlling the starting and stopping of those streams prevents us from doing a lot of cool things so you'll notice when you have a notebook and a single stream

36:45fails we don't know how to restart that one single stream we don't even know what code inside of the notebook started that stream and so it has to restart the entire job when that one stream dies when you switch to using DT we actually we're the ones who called that start we own that stream we know how to start it we can restart we can do fine grain retries we can start a new cluster when

37:09there's an upgrade and get that cluster ready before we stop the old one there's all kinds of things that become powerful and so we really think that for running continuous streaming DT is the right way to do it uh we won't get rid of the notebook way but I really I think that's kind of always been a bit of a hack um and so our goal is to have full parody uh you'll be able to write to different syncs like Kafka you'll be able to manage checkpoints uh and do kind of

37:31fine grain operations on them uh you'll be able to use four each batch even though I think you never should I think it's a very dangerous API um that I see a lot of mistakes made in but I you know our goal is anything you can do instruction streaming you'll be able to do in dld so you can get all of all of those benefits of kind of a declarative runtime uh while still having all the power of structured

37:54streaming and just one thing before before uh Quinton takes the lead if you're new to DLT there is a noome talk um AD DI I mean that was supposed to be Michael and Paul but we had only Paul but it's a cool topic I think it's one hour and a half going through all the introduction of DT the concepts apply

38:16change into I think it's super super important because things people are always get confused when they try to write a DLT pipeline by simplifying sometimes they can get lost but it's cool a session to watch and I'm pretty sure we will get everything you want to know from about DT yeah I was super sad to miss that I unfortunately had covid

38:38but Paul did a great job with it so absolutely look for that talk I think it's a great kind of comprehensive overview of all of the concepts you need in DT all right so so let me do a recap I think you well first of all we like you have to start using DT like because it's so smart and the engine is doing everything for you you still have to Lear Python and SQL because like the a

39:01is not going to do everything for you you have to use serverless with DT because also serverless is going to and not just DLT all of data yeah yeah everything like can I think the best part is like notebook serverless and Warehouse serverless like it's so nice you click it starts it's also super good to do unit test by the way um it's a really good way to run some SQL and see the

39:23results um anything you'd like to add on top of that um you know I think really the only other thing is you know think doing declarative programming is a different way to think and I think you know kind of like Reynold gave some great talks in the past where he showed like Computing something with a for Loop versus Computing it with SQL I think DLT is the same kind of mind switch and so

39:45you know if you if you do try out DLT which you know of course I hope you do do realize it's not a better way to run spark programs it's a better way to do data engineering declaratively and so it's more like writing a squl query than it is like writing a program but once you kind of make that mental switch I think it can make a lot of the tasks you need to do a lot

40:07easier all right amazing thank you so much Michael yeah thank you so much for having me this was great bye bye-bye