Build Pipelines with Spark Declarative Pipelines (Open-Source)

0:05Hey folks, today I have the pleasure to host for the very first time Rayzo. He's a director PM at Databricks. Hi Ray. >> Hey Joseph, thanks for having me here. >> I'm really excited to have you here and I don't want to spoil people but I let you introduce yourself and maybe tell us on which product you're focused. >> Awesome. Hey folks, I'm Ray. I'm on the product team at Databricks primarily

0:28focused on data engineering as specifically the pipelines is something that I spend a lot of time attention on. Before Databricks I spent a long time almost eight years at Databricks on the very early days of cloud and my focus there was on Kinesis. Stream data processing. So you can see the relationship with why I'm driving the declarative pipelines. And so that's me.

0:50>> So I think you guys going to be talking about DLT for declarative pipeline and the new name Spark declarative pipeline. It's D P. >> That's right. It's a formative look of how we change these names but it's Spark declarative pipelines. >> So can you guarantee this is the last rebranding? >> Personally I can. I want to do that. I can but I can't guarantee that. But I hope that is true.

1:12>> Well for those who may miss the info and the news, SDP has been open source and you can now try it as part of Spark 4.1. You can deploy it locally and start building on your ID. I don't know VS code and deploy your deploy your pipeline just to get to get started and I think this is something something that where Databricks invested a lot to make

1:35sure that we still embrace the open source technologies. >> Yeah, that sounds right. And by the way that's actually something we always wanted to do cuz you know as you said the root of Databricks is about open source. But we build open source frameworks, we drive community adoption and we offer something for enterprises and commercialization with better performance, production support as a company. So when we invented uh pipeline a few years back, which was

1:59called the LT as you said. So, that was always a plan. It's just in the early days, we didn't have enough staff to really do the open source push. And also, there was a lot of demand on the product. So, we focused a lot of their resources on really making the product great. And uh now we are at a point where we see a lot of adoption. We have strong conviction on that vision and and the future. In fact, we want all of the data engineering workloads to start to move

2:21from the data frame API into the declarative pipeline APIs. So, now we feel like, you know, the time is right that we should spend energy, spend the resource, give the framework uh into the communities, and we would love everybody to pick up that framework. And all the different vendors platforms should embrace uh declarative pipelines as a new paradigm. And we're very excited about that. >> Yeah, and then I remember I interviewed

2:43Michael in February, and he said at some point we're going to be we're going to open source the LT. We just need to make sure that everything works fine. And yeah, and in June it was announced, and it was uh it was really really great news. I'm pretty sure we're going to see a huge adoption, and it will also help the tools, I don't know, Cloud Cloud Code, Genie Code now to be to write pipelines. And this is something new.

3:06You can use Genie Code to write your pipeline quickly with platform. >> That's actually a very good point, by the way, like on uh you know, things just the era of agents. So, we can talk about that. So, the fascinating fact is that um you know, obviously when when when we invented declarative pipelines back in days, Genie I wasn't quite a thing yet. But the idea of it is actually very simple, which is abstraction, right? If

3:28you look at technological advance, this abstraction happens everywhere. And the core principle of abstraction, or the value of it, is that it makes easier for humans to develop and maintain, to reason about, so that you can use the system in a much easier way. You know, think of SQL, which was a declarative abstraction language to process data. And uh but the fascinating thing is that actually agents likes abstraction,

3:52too. I don't know if you've tried to, you know, interact with the systems. Actually, agents works extremely well with command line tools like CLIs. But if you think about CLIs, it's abstraction, right? It was that somewhat restrictive and more principled interface. Agent can do much better job reasoning about things. So, I think with that, you know, that's another reason by the way like we think the declarative pipeline should be the new norm, the

4:14new de facto for data pipelines is not it makes things easy for the humans, but actually makes much easier for the agents to author these pipelines, reason about the pipelines, and write good quality code, and be able to debug the pipelines under this sort of more declarative interface. You know, I certainly didn't see that coming as one of the reasons, but I think, you know,

4:36coincidentally, things make easy for humans also now it's prime for the agents. >> During the last couple months, a lot of enhancement has been added. Now you do have you have the possibility to copy tables from a apply pipeline to, for example, another lines. You can, I think, more recently announced the possibility to change the system source

4:58without having to run a full recompute in case you're changing the source system, if I'm not wrong. The deep the optimizations are done with the PO predictive optimization. I'm talking about the vacuum, the compaction. You can use liquid clustering with the within your MVs and STs. We have more features on the observability side with the lake flow

5:20with the dashboards that you can deploy and see. I So, is there anything I missed, and where are we going with this with this with the SDP? >> Yeah, so that that's great. So, I think the way I think about the pipeline product in terms of like data engineering workloads and in terms of like declarative framework, essentially, you have two layers of declarative value

5:44propositions. One is at the query layer. So, before you have a data pipeline, you have these spring break it down into each individual queries. Now, what people care about queries is almost always in data engineering is I want incremental processing, incremental computation, and I want to lower latency. In a sense that you don't want to reprocess the data that's already there. And at the query level, there are

6:07two different processing semantics. There's streaming semantics and there's batch semantics. And uh streaming by definition is incremental, right? Because streaming semantics basically means that you keep looking at the source and you process new data as they show up. So, by definition, you only look at new data. Where batch by definition, you look at the entirety of the data in the source, and that's why we have, you know, the technology you've probably heard of called Enzyme behind

6:30Materialized View constructing the data pipeline that handles that incremental computation for you automatically. Now, the truth is that human human brains are still like much better at reasoning about batch because data is finite compared to streaming, which is why I think although streaming's really taken off for very simple workloads on the ingestion side of things. In the transformation line, I personally I still see that

6:53you know, most of the workloads will leverage batch semantics. But Enzyme really makes it simple to handle that incremental processing for you. So, that's the first layer, right? Which is at the query level. So, there's a lot of innovation done in the query level, right? So, we have a auto CDC API on the streaming side of the query makes things very easy um to do type one, type two

7:17tables. There's a lot more innovations happening over there. For example, we have this new API that just GA'd allow you to, you know, in cases you when you don't have the change feeds upstream, let's say your upstream database team, they only give you the snapshot. Now, we allow you to do type one, type two from the sequence of snapshots. We allow very soon we're going to allow you to mix the snapshot with a change feed. So, you can

7:39throw snapshot or change feeds at the same type one, type two table. We're going to handle all the order and logic behind the scenes for you. Then in the in the batch land, we have materialized views, right? Which is again, like it's very Go ahead. >> No, I think it is worth mentioning that with Auto CDC can be done in with five or six lines of while if you wanted to implement such a solution, you would

8:01need at least maybe 200 or even more. So, it makes things way easier for you. >> 100%. And by the way, that's you know, one of the most popular features inside the Cloudera pipeline. And we actually plan to deploy that capability across the Databricks platform, including DB SQL Data Warehouse, including notebook products so that you can use a very simple single statement with a few lines of a

8:24code to do that. Um yeah, and then come back on the materialized view side, people love Enzyme. Uh and materialized view actually guarantees the accuracy of the data based on the source table. So, a lot of people use materialized views for like a gold platform and layer and for query acceleration, dashboard. We are actually adding a new capability, which is coming very soon, which is what we call a replace where. So, instead of refreshing

8:46the entire table to guarantee accuracy, we allow the users to pass a predicate. Things like I only want to refresh today's data or I only want to refresh the data from this particular region, from this particular partition. And within that predicate, actually Enzyme will kick in to do the incrementalization techniques. So, personally, I think that'll be a very popular primitive for people to do

9:08large-scale ETL in the bronze to silver layer. Compared to MD, people will continue to use that for dashboarding and query acceleration. So, that's you know, there's a lot of innovation happening at the query level and I kind of, you know, teed up a little bit on the fact that we're going to give a lot of these query level innovations into, you know, the rest of our product stack we have at Databricks so that all these different users can take advantage of

9:31it. Then the second layer, which is one layer up, right? You have these individual queries. But for data pipelines, it's not a single query. It's a sum of multiple queries. And that's when declarative comes in. Similarly, there's value for streaming, there's value for batch. For batch, it's kind of obvious in a sense that, you know, now system figures out which queries to be run in what order. So, you don't need to manually think about I should run this

9:54table, then that table, then update that table. It infers that [clears throat] order and runs that query orchestration for you. And that's what a pipeline does. Pipeline allows you to, you know, put a bunch of these different streaming and batch queries inside it, it'll figure out the orchestration order for you. Then on the streaming side of things, you know, orchestration kind of no longer really matter, right? Because with continuous streaming, all the queries just keeps running. Like,

10:17there's no order in terms of what to run before what. But if if you think about continuous streaming workloads, what matters is really availability, right? The fact that you're running continuously, either it's mission critical or you require low latency on your business use cases. So, you don't want the pipeline to be down, you know, whether it's for your deployment, change of code, or runtime upgrades. And the fact that we have a pipeline

10:39orchestration functionality for continuous streaming means we understand the data flow graph of how things are being run. So, we can do interesting things like, you can, you know, if you want to, you know, update your code, you have now new business logic. Instead of redeploying the entire DAG, the entire pipeline, we can keep everything else running as is, and you just redeploy one specific query. And in case of like

11:01runtime upgrades, we can warm up a new runtime and a new cluster with that entire DAG and state preloaded, and just do a flip. And I think in in certain case, people call it blue-green deployment. You can flip that to guarantee maximum flexibility. So, that's the the second layer, which is the the pipeline DAG layer for both batch and streaming. Pipelines give

11:23you sort of these specific value propositions. So, I guess my point is summed up with the way to think about the declarative pipeline product is that there's query level innovations, value propositions to the data engineers, there's streaming, there's batch, and then there's a bag layer, which is a data graph layer. That's what a pipeline gives you. Again, you know, it applies to both streaming and batch, and when you sum it up, that's what the cloud of

11:45pipelines is. And you know, catered towards both batch and streaming processing for data pipelines. >> And even the ID, the dev experience has been massively improved. Like the new UI is insane. I think it's even much better than the like regular like environment you will use to write your PySpark. You have the tree where you can define maybe your bronze, your silver

12:07layer, and gold layer. You can also add maybe unit testing, and it can be automated like writing unit testing on the top of your pipeline. You can define the dabs, the the asset bundles directly in the in the workspace. Then you can you can write your code, and you can once you like you you can run the dry run just to make sure that you have no error syntax. And what's cool when

12:30you run like or like run your pipeline, you can see the dag being built in front of you. Can see like every single table being populated and the system and like the lineage being created in the in front of you, and then you can click on each table, you can view the sample. This is something we couldn't do in the past. We can view the sample of the data. We can view the data quality. We can do a partial refresh like rather

12:53than doing I don't know, I just updated the a base like table at the end to enrich my my customer data. I changed just this this customer data maybe for example with new addresses, I will just select those two tables to refresh just the two last components rather than having to refresh the entire pipeline. >> That's right. >> I think the new experience is just

13:15insane, so people just need to give it a try. I think it is worth. You need to forget about the DLT experience. Try it. You going to really love it. And now use SDP like embrace the SDP experience. >> Yeah. So, I think I think you made a good point. By the way, the select refresh capability you mentioned is because of the very fact that that's the data graph level of declaration, you know, declarative

13:38paradigm, which is for the fact that we understand the data graph, so the system can do these things for you as opposed to just a black box, you have to restart it and so forth. But, I think you also made a good point, which is apart from these core engine innovation that I just mentioned, it's super important for us to provide a solid dev experience, like the IDE you mentioned, where and by the way, we've, you know, you should try it out if you've tried it before, it runs very, very fast. It gives you the bag,

14:01it gives you the observability, so you have everything in one interface. And you can build your data pipelines with visual. And uh also, um I don't know if you guys have heard about it, that we have this Lakehouse Designer product, which is it infuses AI to give you that visual feedback loop and build these pipelines for you. We also have now data and during agents, so if you pop into that uh editor that Youssef you were talking about, on the

14:23right panel where we have that uh you know, Genie code from Databricks, you can actually talk to it and it'll write declarative pipeline code for you. So, I think about that's another piece where, you know, with GenAI, I think a lot of that development loop using interface will be revolutionized, make it even even more productive. And I think but because of the very fact that uh declarative pipeline with that abstract interface will make that experience much

14:46smoother compared to like generic coding experience. So, that's one on the dev side. Then on the other side, which is observability, right? Like, you know, for all the data engineers out there, you kind of know, I don't know, you probably spent less time on writing code versus make pipeline, you know, just keep running, right? Because the schemas might change, business requirements change, you need to backfill the data, things arrive late, you have to deal with all that stuff. And observability

15:09experience with um system tables, event logs, and you know, UI, run history, these are absolutely critical. And in my opinion, by the way, that's another area where Agent can play a huge role. The very fact that uh you know, the flavor pipelines, we have jobs of productionization, we have UC, you know, the catalog store the lineage information. I think with all that data,

15:32it makes very straightforward for the agent to reason about when your pipeline fails or have warnings, like what is going on and give you recommendations. So, that's another area actually we're looking at very very closely, which I'm very excited by. You know, I think you know, because of the declarative nature, the agents can fundamentally change how people develop their pipelines and how people maintain and keep their pipeline up and running. But stay tuned. I don't

15:55want to spoil it, but we'll probably have something to show you at the Data and AI Summit this year. >> Yeah, and and I think you mentioned something very important about Genie code. Why should I use Genie code over, I don't know, cloud code or open AI codex? Well, with Unity catalog, we have the metadata, we have the context, which means when you write the code, it automatically understand what which

16:18sources are we using. And and when I'm developing my script, if I have an error, I don't need to copy paste the error in the another tab to make sure that I fix the issue within the Genie code, I can automatically click on debug the error and it will it will trigger. Same thing if I want to use SDP, well, the Genie code already integrate the data engineering agent, which understand

16:42and knows the system or like the syntax in order to write SDP whether using Python code Python or SQL. And on the top of what Array mentioned, we have the observability. So, all the logs, everything happening inside the SDP are stored in I mean, you can define the data table where those logs are stored and then you can build an AI BI

17:04dashboard. You can also use Genie code. Genie code ask the authoring authoring agent that will create a dashboard using those those logs and then you'll be able to monitor Databricks, to monitor SDP with Databricks. And you can have this full observability plus the other dashboard to track the cost. >> Yeah, I think that's spot on. In fact, I think under the hood it's all running.

17:26Everybody's all running the same foundational models. So, I think really the key is that the contextual There's two things. One is the contextual information, as you mentioned. It understands the system with the context, with all the metadata and information in Unity Catalog and all this other ability. Then the second aspect is the native like part of the integration. fact that you can make this super easy. Like, you can write a code, you can debug, you can fix

17:48it, you can rerun, take me to the observability, like all that coherent experience that we provide. In fact, actually by the way, the other thing I kind of want to mention is that uh we actually develop a lot of the skills to make the model being able to write pipeline code very very easy. We've actually We're going to as a matter of already I I don't know the specific pipeline, but we plan to make all of these skills available uh for the

18:11entire community. So, even if you're not using the Databricks IDE, even if you're using Cursor or Cloud, all these things, we will you will have these skills to be able to easily debug pipelines. Uh but again, if you use Genie code, you can get a much more native experience, but we also want all the other IDEs to be able to easily build these pipelines with the skills we've built. >> Yeah, and I think if I'm not wrong, even Databricks is relying a lot on SDP. For

18:34example, if you have a lake if you have a Delta table and you want to create a lake base, like the sync tables are using behind the scenes SDP. >> That's right. Yeah, the Databricks >> If I'm not wrong, is you also using SDP? >> Yeah, and and you know, if you look at the materialized view capabilities in the data warehouse and very soon we're going to add more, like as I said, auto CDC, replace where with ends and

18:56incremental organization and enables, all that stuff is running SDP behind the scenes. The way we sort of look at it is that, you know, before AI now with AI even more so, data flow workloads is sort of ubiquitous, right? No matter what you do do, you you need the data. You know, you need to process data, you need to move the data. And a fundamental primitive is actually the same, right? I'll give you another

19:18example in a case that now a AI BI dashboard, we actually created [clears throat] materialized views behind the scenes to cache and, you know, for better performance. Which is why I think, you know, the fundamentals of the as I mentioned the first query layer, whether it's streaming, whether it's batch, that incremental processing, that's offered by the Delta pipeline is ubiquitous. I can see that being used

19:40across the board uh for AI applications, for data applications, and internally it's powering a lot of our own products. >> And I think also uh lately you released you released like the compatibility mode, and I think these features can also be very very helpful for those who can read streaming tables or MVs outside Databricks. Like compatibility mode can be very helpful

20:02because it will create a Delta table that can be read using any uh Delta Delta forgot the reader and writer. No, like Delta reader and writer, like clients, any Delta clients. That's why I was looking for the word clients, any Delta clients. >> So uh Iceberg clients. I don't need Delta, but also Iceberg clients. >> Iceberg clients. >> I I can ex- explain why about

20:25compatibility mode. It's come back to the fact that, you know, we're open company. Our root is an open source, and not about only about the processing engine, but also on table formats. We want the table formats to be open. We want all the engines to be able to interoperate uh on the same data sets across the board. Uh but the very fact is there's, you know, one catch to that, which is at the same time for our managed table and materialized views. I think you talked about some of the

20:47things on like, you know, predictive optimization and uh liquid clustering. And for materialized views, we do this incrementalization under the hood, which means that it requires the system to manipulate the data in certain ways to be able to do these optimization layout and read and write performance for the for the readers. But the very fact is that then you require the reader to be able to process this unique information

21:10so that you know that that users still get the data they want, which is why we spend a lot of time to work on this compatibility mode and there are actually two. So you can either do the plain compatibility mode which will create a new plain parquet file for readers that absolutely cannot understand modern protocols. But for any new modern readers, whether it's iceberg

21:32or Delta client or readers, we just released a feature that allows you to read all of the managed tables including materialized views without a a clone of the raw parquet file. So basically with that we give the users, the customers the best of all the two worlds. One is like all of the table is optimized, incremental processing, good for read, good for write performance. But at the same time you still get that interoperability and open access to the

21:55table. That's the spirit of uh the core of the compatibility. There was a lot of actually I was involved in that. There was a lot of technical challenges to give people both of the worlds, but but we did it. The team pulled it off. So I'm very excited. >> I think you've summarized everything. So I think the only thing that's I'm missing right now is uh an SDP uh swag.

22:17>> You have a you need a catalog winner. We'll figure out a winner. I'll get you SDP swag for sure. >> That >> Actually our swag was still on the old uh sorry on the final naming convention. >> Thank you so much Ray. It was very interesting. Uh thank you so much and for those who want to know everything about SDP, make sure to register for the online for those who can't go to San

22:40Francisco to register for the online uh data and AI summit experience. You would be able to view the in the live the conferences or a live broadcast and you can also later have access to all the sessions that will be uploaded to the data and AI summit uh website. >> Awesome. That's great. Thanks for having me here Asif. It was great chatting with you.

23:02>> Likewise. >> Awesome. See you, man. >> Lake Tahoe by the [music] >> [music] [music] >> Drop it.