Executive Interview
Future of Data Engineering: 2025 & Beyond with Michael Armbrust
Michael Armbrust · Distinguished Engineer · LinkedIn
instead of coding in python or Scala you're going to be coding in English you try it you know it's not working you change the query try again the test works and like it's super fast so you'll
say I want to copy data from there to this spot in the catalog and the system will take care of it for you we're simplifying more and more this user experience there is no migration there's
no breaking changes you'll get incremental improvements to the existing products as you're using them today hi folks hi Q uh today we're hosting Michael armur at distinguish engineer at data bricks he designed
spark SQL then streaming then Delta and now he's focusing on the next generation of data engineering and my question is is there anything Michael doesn't do you're too [Laughter] kind all right so um maybe to to get it
started I wanted to Michael to talk about the like the overall data engineering ecosystem like Spar is um I think it's 10 years old maybe a little bit more little more I think yeah yeah I
was wondering like what do you see in the market like uh nowadays if you are a new data engineer which tool would you be using like spark but do you do also python do you do do you do Scala I don't think so
anymore really maybe sequel to use AI to build everything I don't know what what how do you see the market yeah I mean well I think if you look at even just the history of spark we have been slowly
upleveling spark over time so you know like before spark there was Hadoop and you had to spell out every little detail you had to have these different like output writers and input wrer readers
that would control exactly how the data was serialized you had to write out your algorithm in you know full detail and Spark made that a lot easier it gave you these like simple functional commands uh
where you could map and reduce and do this stuff but all in just like a few lines of scholar code um and then we added spark SQL to it and like that was a big simplification where now you could
use data frames in Python data frames in Scala or just plain SQL commands to run it and so I think this this trend will continue to have kind of higher level abstractions on top of it and I think
you know really where we're probably heading is instead of coding in python or Scala you're going to be coding in English you will talk to an llm in English which is this like messy uh
ambiguous language and the llm will produce python Scala SQL one of these unambiguous languages um and that will be that'll kind of be how you're working and I think I think that's good that
that Trend upward is going to continue and what what's the big chunk of um like language you see running on the bricks is it SQL or is it python um I mean you know I think I think that it's pretty
split but definitely SQL and python are the ones that are growing and schola use is declining over time and I think it really between SQL and pyth Pyon it's more of a Persona thing it's really if you know SQL and
that's your preferred tool of choice you like to you continue coding in Sequel if you want to do more procedural coding if you have like a bunch of steps you want to orchestrate them together and you
want to modularize your code uh and you want to interact with libraries like pytorch or these other things then then you're probably using Python and DLT uh was introduced two years ago and can you share a bit on the
story around DLT so you know yeah DT has come gone through quite a few Evolutions over time but really the goal was exactly what I said at the beginning we want to keep upleveling the task of doing data
engineering so you know when we were building spark SQL we were building this very generic tool for executing declarative SQL queries and the magic trick of that that declarative language
is instead of saying like in order to compute an average I will first do a sum and I will do a count and then I will divide them you just say give me the average and the system figures out the
right way to do it where the shuffles need to go uh where it's going to do partial aggregation all of those different steps just kind of happen automatically DT is the next level where
we want to make the entire process of doing data engineering declarative so now there are actually verbs for create this table and create it up to date uh and keep it up to date or do change data
capture and handle all of the outof order data and keeping history and all the other things you need to do you just say what you want to have happen and the system will figure out how to do
it um it started off as you know if when we first created DLT the product name was probably going to be serverless streaming the idea was just take spark streaming and make it easier to run add
all of the retries and the metrics and the auto scaling and all the things you need to do to run a a streaming application without having to tune a whole bunch of knobs but what we quickly
Learned was continuous streaming is a tiny fraction of the data engineering people want to do it's great when you want low latency but it's pretty expensive um and also streaming has this
this fundamental it's both a limitation and a superpower streaming promises it's only going to read the input data once and so the types of things you can compute with it um are limited but also it's
guaranteed to have a certain performance and so over time DLT evolved from this kind of very focused on streaming pipelines to a generic tool for doing data engineering but declaratively yeah I feel like it's a
like we we say streaming but it's more incremental Pipeline and then you can decide uh like the latency do you want it running every day or like every second it's up to you but like the key
is an incremental incremental pipeline that's exactly right yeah that's one of our magic tricks when people hear the word streaming they often think Super complicated super expensive in order to
get the lowest latency and Spark can do that and DT can do that especially with the Advent of real time mode which is something that's in private preview right now where we actually now able to
run continuous streaming pipelines with subse latencies which is you know competitive with some of the best streaming platforms that are out there today um but the really cool thing about
spark and the thing that nobody else kind of in the streaming space can do is you can tune that cost latency trade-off to me streaming does not mean necessarily low latency it can mean that
but it's exactly what you said it means incremental computation it means reading the input only once and it turns out a lot of data engineering is exactly that when you're ingesting data you you only
want to read it once you know when you're doing Transformations you only want to read the input once and with triggered streaming inside of uh you know DLT we could do that super easily
and the best part is unlike when you were using kind of plain spark when you go back between triggered streaming and continuous streaming you don't need to make any code changes so if your
business requirements uh change over time it's one click in the UI to change it to a a continuous streaming Pipeline and get those super low latencies so so doing incremental like if it's new data and you just want to
open the data to an existing table like I can see how it's kind of simple you just consume the new data and you open that to the table but in most of the pipeline we have uh like you want to do
something else you want to do aggregation maybe you want to join with other tables creating views and stuff like that and I think that's a really hard stuff because most of the time you
if you do it yourself you would have to recompute everything and I know DT has some magic behind the scene to you know to help with that can you can you share and tell us how it's working yeah
absolutely and this is I think one of the biggest challenges people have when they're understanding DLT is like DT has several different abstractions it has tables that have streams writing to them
uh and it has materialized views and really the tradeoff between these two abstractions is like I said before streaming has this limitation and this superpower that it only reads the input
once and that limits the types of queries you can write so you can do aggregations but you can't really do aggregations where you need to keep arbit amounts of State in memory so for
example if you wanted to do like histograms over arbitrarily large data sets streaming is not a super great thing for that you'd want to use like a sketch and do an approximate histogram
instead you want to do uh time window averages where the data arrives in a bounded amount of time streaming is a perfect case for that would you want to do outer joints window functions these
more complicated things then materialized views are the right solution materialized views are still incremental in many cases and as we improve the the enzyme query Optimizer that's our that's kind of the new system
we put on top of catalyst that not only understands how to optimize queries but knows how to update existing results to queries efficiently when the input changes uh but uh enzyme can run all
types of queries it can do joins it can do outer joins it can do aggregations and the magic trick is it's allowed to read the input again if it needs to so let me give you an example of where that
would be necessary if I'm Computing a Max and the maximum value is deleted from the input table well now I have to go back and look at the input table and find out what the next maximum value is
that's something that streaming could never handle it can't do retractions it can't understand the idea of data disappearing because that would require sometimes arbitrary computation
materialized views will automatically take care of that so in the cases where it's possible it will be as fastest streaming it'll do exactly what you said it'll read only the new data append it
Downstream everything will be fast but in the cases where something changes or in the cases where there's an expensive uh operation that requires extra work materialized views will also do that
trans parently so there's this tradeoff between guaranteed performance and expressive power and what that means is you can really kind of no matter what your requirements are you can get DLT to
do them you just need to kind of think about what's more important to you okay and like DLT like in the past we could not choose any like what we call the run times you just need you just choose
let's say the the uh the number of workers and the type of machine and that's it now we have uh server list which also gives like simplify more the uh deployment of DG pipeline but can you
tell us more how does DT work with serverless and how this can help the customers optimize their cost but also the performance yeah totally so serverless is you know not only a DLT
feature it's something we're kind of doing to simplify the entire data brick platform and the nice thing about serverless is you no longer need a cloud account you no longer need to worry
about IM roles or all of kind of the setup of of you know whatever whatever Cloud you're running on we take care of that for you and it has a couple of magic powers so one is you can now sign
up for data brecks with only a credit card and go all the way to processing massive amounts of data sets without managing any other infrastructure you don't need to worry about quotas or any
of that kind of stuff um but also serverless has a couple of magic tricks it's got a super awesome autoscaling algorithm that works not only in DLT but across all of the other products and it
has and the one of the kind of cool parts of of serverless is we only charge you for the compute that you use so it's on us to figure out how to allocate those machines and how to bin pack them
and do all these other things and you're only ever paying for the capacity that's actually doing useful work for you um serverless also has a bunch of performance features since we manage the
entire stack we can do things uh more quickly in some cases so especially with streaming workloads we can pipeline different uh we can do streaming pipelining and execute multiple micro
batches in parallel we can automatically tune the admission control parameters things like Max files per trigger since we know exactly how big the cluster is and how much extra how much extra space
there is and how much space there is in the warm pool we kind of we take care of all of that automatically and so in some cases we've seen that serverless can be both faster and cheaper for large scale
SK streaming ingestion jobs and simpler so so we have we we have the LT and webs you can use DT like as a as a product and create your DT pipeline but then you can also create create
materialized view which I think are using DT kind of in the backand uh I know also when you create Victor search for example we using DT in the back end how we are we starting build to build
everything on top of DT like as an engine and then offering higher level um function to customers is that how it's you know working yeah so I think the right way to think about this is you
know DLT is like it's it's a brand name and it's an enabling technology and what DLT that that engine that kind of that runtime is good for is it's good for doing data engineering it's good for
reading data from one place transforming it and writing it to another place and it simplifies a lot of that work and of course even some of the products that we do internally have that exact same
problem so uh system tables are built with DLT we do our own internal security with d LT when we're moving data into an online table or into a vector a vector index DLT does that movement when you're
ingesting data using um lak flow connect um that also uses DT um but really the abstraction when you when you know you as a customer are using DT there kind of a few things you need to know about you
need to know about streaming tables and materialized views which we talked about a little bit uh you need to know if you want to do Advanced things you need to know about flows which are actually like
it's it's streaming but it's also things that aren't streaming like for example doing onetime backfills so if I want to have a single table I want to do a one-time backfill and then I want to
read from two different kof in different regions and WR into that table then you're going to start using flows and then the final DT abstraction is a pipeline so streaming tables materialize
views you can use just on their own you can just go into DB SQL say create streaming table say create materialized view and the system will automatically manage everything you don't need to know
any other Concepts just one one line of SQL can build a highly scalable ingestion pipeline that's automatically managed but when you're doing like a large data engineering task you want to
do the kind of full software development life cycle you want have source code that defines all of the tables you want to Version Control it you want to deploy all of those assets together and that's
where pipelines come into play a pipeline is just a collection of streaming tables materialized views and flows that are deployed and executed together and so just think of it as like
a a management abstraction on it so kind of just to summarize DT is this enabling engine that lives under the covers the important things for you to understand are these streaming tables materialized
views and pipelines yeah and since you mention um like mvs and STS um when when it comes to developing code I know it can be it was it was in the past a bit tricky to debug the the code when When developing
on DLT and I've seen that we are bringing we're simplifying more and more this user experience but can you tell us more on this uh on this part yeah totally so I think one of the first
steps we did was these DB SQL materialized views so now you don't need to do any extra setup you just run a single SQL command it runs like normal and it creates a DT object but even for
more complicated pipelines we're spending a lot of time on the developer experience this quarter uh we recently moved we recently decided that basically this is one of the we really like this
underlying technology but we think the user experience of doing development has been holding us back a little bit so we recently did a pretty big reallocation of Engineers to get a lot more people
working on improving that developer experience and there's a couple of things we're working on so one is DLT has this magic power called analysis because DLT is declarative we can actually analyze all of your code
without actually running it so we can look and say are all of the tables in this pipeline valid are all of the column names table names data types functions correct without actually doing
anything to the production data and that's something that's really only possible when you have a fully declarative language you can't do that with something Turing complete um and so
you can so basically what the lets us do is when you're what we want to do is when you're editing you can run these validate commands but we want to make that validate operation so fast that it
can be continually running in the background so we have a bunch of benchmarks that we're doing right now we basically once a week we're reporting all the way up to Ali our CEO exactly
how long it takes to analyze a single table how long it takes to analyze python versus SQL how long it takes to analyze a table a pipeline has 100 tables in it and we're going to spend a
lot of time optimizing that so doing things in parallel skipping steps where nothing is changed uh removing redundant work that we're doing and we want to get that to be super fast and I think what
you're going to end up with is something that looks a lot like a cloud ID for doing data engineering um and I'm pretty excited about that because I think it's it's I think we can actually kind of you
know today people are very used to notebooks and there's this really nice interactive experience but also you have to make sure that when you run those notebook commands you're not breaking
production and I think that's like that's a that's a trick that's like that's a a pretty dangerous thing and maybe not the best way to do it's a great way to do I think data science but
it's not a great way to do proper software engineering and so I think with this new investment in the development experience in the data bricks workspace we might actually be able to Leap Frog
notebooks in terms of the what it's like to do development of production data pipelines and and what about the unit test I think unit test has been like a challenge forever in data engineering
because you have to bring the data you have to test the transformation the input the outputs yeah that's a great question you know and it's actually something that we spent a bunch of time
prototyping when we built the first version of DT you know like the first version of DLT was this thing we like I built on my laptop that just reached out and used datab bricks apis to like run a
job um and that first version had this really cool unit testing framework because again another magic trick of declarative programming is we can substitute things in the graph so we can
say take this data flow graph that you've defined this table reads from this table reads from you know wres to this table um and we can say well actually that table is the you know the
production data let's substitute in this sample data instead and see what result comes out at the end so we don't know the exact shape of this but we do really want to build data tests where you can
say given this input make sure this output exists and have tens or hundreds of those examples that the system runs automatically for you yeah that would be awesome and if you can have that and and
I don't know kind of live trying it you try it you know it's not working you change the query try again the test works and like it's super fast I could see how it could speed up everything
definitely and I have a tough question I think this one cames a lot uh is DLT going to be open source at some point yeah so you know data breaks has a long history of doing open source we've
started around uh you know this open source project spark the first major project I worked on at data brick spark SQL I remember the day that we opened that like I think it was 11,000 of code
PR against uh Apache spark to kind of Open Source that as well um Delta was open sourced um so you know we absolutely want to open source DT it's just a matter of engineering uh we need
to make sure we have the right engineering capacity and right now our focus is on making sure the experience inside of data bricks is as good as possible but we really like to work with
the Apache spark Community we think that this is a fundamentally better way to write data engineering programs and we' love to make those kind of the standard apis and Spark for doing it and have
some kind of simple Runner and open source that could run DT programs as well um so stay tuned it's something we absolutely want to do but right now our focus is on making sure the experience
in data breaks is as good as possible the one other thing I'll say is the most important parts of DT are already open source the data that DLT produces is stored in Delta and is available in
unity catalog that's those are both already open source the Transformations you're writing the nice thing about spark is it's not the nice thing about DT is it's not actually a new language
for writing Transformations it's just spark SQL it's just data frames and to me the data and the Transformations that have your business logic that's the most valuable part that's the part that would
be very expensive to port to another system uh but we want to make the whole all of it all of it open source including the runner it's on white and I see also often customers like asking question about DLT uh and
and DBT uh but I feel like DBT is just using another Hood the DT computation at the end uh but maybe can you can you explain us how it's working like and and as a user like do you have any Lo in
starting to use DT or DBT and how how you should be thinking about it yeah so I love DBT you know I think it's a pretty I think we have very similar goals between the kind of the DT project
and the DBT project this idea that rather than managing your data infrastructure as a bunch of ad hoc commands you you run in production that's just not the right way to do it you want to have a Version Control
description of all of the tables that are going to be inside of your catalog and you know DBT and DT are kind of very similar in that way um the key difference is you know DBT runs outside
of the database engine so it basically you know it does like ginger templating to build SQL queries that it then executes against some other engine and that limits what they can do you can't
it's difficult to incrementalization or things like that um but fortunately you no longer have to choose uh there's a data bricks connector for DBT or the datab brick adapter um and it creates
two new materialization types streaming table and materialized view and so if you are already familiar with DBT you should absolutely keep using it um all you need to do is use those
materialization types and you get can get all of the power that is only possible when you actually own the entire compute engine um and if you're not using DVT you know I do think DT
kind of solves a lot of the same problems it makes it possible to kind of declaratively specify an entire graph of dependencies and automatically keep them up to date that's perfect and I think now we
will talk about Lake flow which also is going to be built on the top of DLT if I'm not wrong so can you give us an overview on what lake flow is yeah so you know Lake flow is going to
be the next evolution of the data engineering products inside of data bricks it always used to be that you know there was Data bricks jobs or data bricks workflows there was DT and they
were seen as these kind of very separate entities and um you know part of that is one of them is they're just very different ways of writing a notebook to do data engineering versus writing DT is
kind of a different way of thinking but really I think that that that dichotomy is wrong um and the the idea of Lake flow is we want to evolve this so that it makes it very easy to use the right
tool for the right job and so the way I look at it is there's data bricks jobs uh and which are you know now going to be called Lake flow jobs so really DLT or lak flow is the evolution of all of
these Technologies it's not just DT it's workflows DT uh Lake flow connect all of these things kind of wrapped into one coherent product and so the way I like to think about it is there's you know
Lake flow jobs and you should always be using that for production a job is the way that you run something reliably either on a schedule or when some other action happens like a table gets updated
a file gets uploaded some other job finishes you should always be using data bricks uh Lake flow jobs for that then there's notebooks notebooks are great for uh doing data science and
investigation you should continue to use them that way notebooks are also a great way to run Legacy spark code if you've got data engineering that's already working and you don't want to change it
it absolutely that's the right way to run spark programs and I use the word program here kind of very specifically when you have procedural logic that says if this then that uh you know o
overwrite this partition where you're actually telling the system how that data engineering should be done you should continue to use notebooks and you should continue to schedule it with L
flow jobs and then what we're what we're doing with L flow is we're now moving DLT kind of into this same UI so that you can seamlessly create materialized views streaming and pipelines all within
that same experience so you know often your data engineering job might actually have all of these aspects if you want to start by downloading data from an FTP server well that sounds like procedural
logic so you should use a notebook for it and then if you want to ingest that data into a table well that sounds a lot like a streaming table with autoloader you should use streaming tables for it
and if you want to transform that data to build aggregations uh that are going to power a dashboard you should use a materialized view for that and a dashboard and you can put that whole
thing into a lake flow job and and what about what about AI because I've been recently I've been using cursor AI uh like to do all my python stuff like it's so amazing like I'm just
you know giving a few instruction like I got the python it's up and running even SQL uh are we going to inject a lot of AI within leg flow so that you can super eily build your pipelines and set up
everything yeah absolutely so you I think the AI is already pretty good at writing SQL and it's pretty good at doing spark data frame operations and so those parts should work seamlessly
already you know inside of Lake flow uh and whatnot um what we're working on is you know it turns out these large language models were trained you know on corpuses that were collected several
years ago before the DT syntax existed and so we're working really closely with the Mosaic team and the data BRS assistant team to uh make sure that we train it on this new syntax that we have
produced so they can provide good answers for that and that's actually kind of an effort that that just started this quarter to really come up with a regiment for measuring how good is it at understanding our
specific syntaxes that we've added on top of spark um and how often is it giving good answers there and how can we improve that with more training I I can do you think like if you are a new data engineer do you still
want to learn how to do python or do you think like in a few years it's only going to be you know write some English generate some you know Automation and that's it well it comes back to kind of what I said before you
know English is this messy ambiguous language whereas Python and SQL have well- defined semantics you know exactly what they're going to do so I think it's never a problem to be able to read and
understand that ambiguous thing I think that will continue to be a valuable skill because you'll be able to say is what the AI produced correct um but will you be writing a lot by hand I think
that's going to be less and less I will say even in my own like stuff at work or the stuff I do in my free time I write less and less codee I let the AI write it I look at it I tell the AI how to
correct it and then I then I use it um because there's just a lot of tedium in programming that goes away when the AI is writing it for you yeah it's super super interesting like and by the way things you mentioned
about the data breaks assistant and DT this something we talked about recently with Weston is leading the data breaks assistant team and he told me about this U this new project to make sure that DT
uh is well un understood by the the assistant I was just curious to understand uh we did acquire Aron or Arion I'm always struggling to AR I asked the founder the authoritative pronunciation okay so for um how uh will
this integration of archon works with Lake flow because I've seen like Arun used to have a lot of connectors and now we're deploying um I think now we have uh I think maybe service now will come
next quarter we Google analytics Salesforce and more so how does work this integration totally so all of this will live under like the lake flow connect umbrella and just you know for people
who haven't heard of it Lake flow connect is going to be a no code pointand click ingestion experience built into Data breaks so you'll say I want to copy data from there to this spot in the catalog and the system will
take care of it for you it uses DLT under the covers but you don't need to know anything about DLT in order for this to to happen um the archon acquisition was super valuable for a key
piece of this which is getting the change log from these different databases so DT has always had this pretty cool technology called apply changes into where given a change feed that has a primary key and some ordering
we will create a table that has the latest upto-date copy no matter what order that change feed arrives in even if parts of it are than others and a bunch of bunch of kind of complicated
distributed systems programming stuff there um but what we didn't have the part that we always left up to our user was how do you get that change feed and that can be super difficult you have to
pay for Golden Gate you have to set up open source projects like deum you have to figure out a way to like get them up into the cl to get the results up into the cloud with Kafka storing the data on
S3 you have to build an ingestion pipeline to pull it in and what we're working on right now is taking that super awesome technology that we got from AR on and building it into the
datab Bri platforms that we can handle that whole step you point us at the database we get the change feed we run apply changes and we create those tables for you and we'll be able to do this
whether your database lives in the cloud that's in uh you know in private preview right now for SQL Server already or on Prem we'll run what we call the ingestion Gateway inside of your data
center you know behind your firewall and it will upload only the data you want into a secure location and unity catalog where we will then apply app it into a Delta table um so archan is the thing
Aran is doing is it's giving that that change feed uh you know for all of those all of those databases and then alongside of that we've been building connectors for a lot of SAS services so
uh service now Salesforce Google analytics uh SharePoint Google Drive whatever it is we want to build a connector for it if we don't have it talk to your account team and let us know because we are we're rapidly
building that out you know it took us a while to get the infrastructure in place um and now that we have a couple connectors out there I'm really excited to kind of start doing the like breath
for search of the space uh really building all of those out and Michael one question so the inis pipeline that we have to do all the CDC and you know ingesting all the changes is it running
uh with DT I mean under the data bricks yes yeah it's all running on top of DLT like I said you don't need to know anything about DLT in order for this to happen uh but the nice thing is if you
are familiar with DLT and you want to do monitoring uh it's the same event log it's the same stuff so if you are a more sophisticated user that's already familiar with this part of it you can
absolutely take advantage of that and have one single way to think about how is my data engineering proceeding what's data quality look like all that kind of stuff but if you're uh you know just a
you know if you're newer to our platform just learning SQL or you're even just using the assistant lak flow connect is for you because you don't need to do any programming you don't need to know
anything about pipelines or streaming tables or any of this stuff all you have to do is authenticate with the sources system and we'll take care of the rest and if I'm not wrong system tables is
all is also using behind the scene DLT and then sharing the results with um using Delta sharing between our control plane and the uh customers workspace yeah that's right these are you know
these are like implementation details but I do love that we eat our own dog food and use our own software in order to do it uh you know it turns out data bricks is actually one of the biggest
customers of DLT um and yeah absolutely some of the system tables are produced with it and like you said yeah we use Delta sharing to share it securely with our customers um so always always fun to
see technology Reus like that and you mentioned at the beginning that all our offering ser and that's that's that's true of course but is Lake flow going to be mainly serverless or not well you
know I one of the key things of lak flow is we really want to evolve the existing products I know that like moving to to Unity catalog and other stuff like that those are big migrations for customers
migrations can be painful um and it's super important to us that with Lake flow there is no migration there's no breaking changes you'll get incremental improvements to the existing products as
you're using them today um but a lot of the goodies a lot of the things that make Lake flow super easy to use will be things we can only do in serverless when we control the entire stack so um even
if you're not on serverless uh you know you you will still get a lot of benefits from lak flow but in order to get the simplest best experience you should absolutely be looking at getting
serverless I think it's really a game Cher in the things it allows us to control and the things that we can control are the problems we can take away from our users and so they don't
have to worry about them anymore and they can focus on what you know uniquely they know about which is their domain and how to get value from their data that's that's right and I've seen I
think a lot of in I think a lot of several features are coming next quarter if I'm not wrong we'll have the possibility to write to multiple schemas we will be able to get rid of the live
keyword and yeah so you're you're talking about direct publishing mode which is the weird internal name for what I would call making DT simple so you know when we when we built DLT we
spent a lot of time looking at dbd and and other projects like that and dbd had this way of ref referencing dependencies that were inside of the same graph so you would say like you know in DBT it's
a ref uh where you say ref give me this table and in DLT you would say live dot give me this table and it would kind of find the right version of that table but what we realized is that was a we talked
to a whole bunch of users that was confusing them it looked like you know they didn't want to learn some new thing they wanted to just use SQL um and this also came with a bunch of restrictions
if if a pipeline is targeted at one specific schema well can only rate to that schema and so the thing that we are private previewing today is a dramatically simpler way of working with
DT so it's basically it becomes just SQL we we realize that we own the entire engine and so and we own the entire catalog so we don't need these extra affordances to tell us where to look for
the table we can just look up the table in the catalog so at direct publishing mode so first of all no breaking changes your existing almost all DLT code will continue to work we'll just ignore that
live keyword um and what the the stuff that is coming is you'll be able to write SQL just like SQL and it will follow the exact same rules as if you were working in a notebook or in DB SQL
so there a pipeline will have a default catalog and schema if you don't if you just say select from table those defaults will be used or if you say select from schema. table that default
catalog will be used um and same thing for when you say create materialized view the default catalog and schema will be used um you could just reference tables if they're in the pipeline we'll detect the
dependency if they're not in the pipeline we won't we'll just read it from the catalog like normal um and finally if you want to write to multiple schemas all you need to do is save what
those schemas are so if you say create materialized view A.B.C it will create the materialized view in that location and you can use all of the metaprogramming and parameterization
inside of DT to kind of control that and do cool things to have a separate development copy versus your production copy and all that kind of stuff yeah and I've seen also the cool stuff about the sync also I mean I've
been following the road map for for a very long time no these are good questions you know and the sync is part of a more General effort our goal is to have full parody in what you can do in
structured streaming in DT you know like I think notebooks are great for doing kind of like I said procedural data engineering but streaming has all always been pretty declarative the only part
that is not declarative in streaming in a notebook is calling do start and you controlling the starting and stopping of those streams prevents us from doing a lot of cool things so you'll notice when
you have a notebook and a single stream fails we don't know how to restart that one single stream we don't even know what code inside of the notebook started that stream and so it has to restart the
entire job when that one stream dies when you switch to using DT we actually we're the ones who called that start we own that stream we know how to start it we can restart we can do fine grain
retries we can start a new cluster when there's an upgrade and get that cluster ready before we stop the old one there's all kinds of things that become powerful and so we really think that for running
continuous streaming DT is the right way to do it uh we won't get rid of the notebook way but I really I think that's kind of always been a bit of a hack um and so our goal is to have full parody
uh you'll be able to write to different syncs like Kafka you'll be able to manage checkpoints uh and do kind of fine grain operations on them uh you'll be able to use four each batch even
though I think you never should I think it's a very dangerous API um that I see a lot of mistakes made in but I you know our goal is anything you can do instruction streaming you'll be able to
do in dld so you can get all of all of those benefits of kind of a declarative runtime uh while still having all the power of structured streaming and just one thing before before uh Quinton takes the the lead if
you're new to DLT there is a noome talk um AD DI I mean that was supposed to be Michael and Paul but we had only Paul but it's a cool topic I think it's one hour and a half going through all the
introduction of DT the concepts apply change into I think it's super super important because things people are always get confused when they try to write a DLT pipeline by simplifying
sometimes they can get lost but it's cool a session to watch and I'm pretty sure we will get everything you want to know from about DT yeah I was super sad to miss that I unfortunately had covid
but Paul did a great job with it so absolutely look for that talk I think it's a great kind of comprehensive overview of all of the concepts you need in DT all right so so let me do a recap I
think you well first of all we like you have to start using DT like because it's so smart and the engine is doing everything for you you still have to Lear Python and SQL because like the a
is not going to do everything for you you have to use serverless with DT because also serverless is going to and not just DLT all of data yeah yeah everything like can I think the best
part is like notebook serverless and Warehouse serverless like it's so nice you click it starts it's also super good to do unit test by the way um it's a really good way to run some SQL and see
the results um anything you'd like to add on top of that um you know I think really the only other thing is is you know think doing declarative programming is a different way to think and I think you
know kind of like Reynold gave some great talks in the past where he showed like Computing something with a for Loop versus Computing it with SQL I think DLT is the same kind of mind switch and so
you know if you if you do try out DLT which you know of course I hope you do do realize it's not a better way to run spark programs it's a better way to do data engineering declaratively and so
it's more like writing a squl query than it is like writing a program but once you kind of make that mental switch I think it can make a lot of the tasks you need to do a lot easier all right amazing thank you so
much Michael yeah thank you so much for having me this was great bye bye-bye