Structured Streaming for Real-Time Data Engineering

0:04Hi, if you are a data engineer, you probably spend most of your time building standard batch pipelines and nightly ETL jobs. It's predictable and it works. But the moment your business demands real-time data, that entire setup changes. You aren't dealing with finite file anymore. You're dealing with endless data streams, out-of-order events and infrastructure that

0:26absolutely cannot go down. Today, we're breaking down the exact step-by-step roadmap by transitioning from batch engineering to real-time analytics without, of course, breaking your production systems. To guide us through this progression, I'm joined with Subhodip and Harsha. They are They work both of them for Databricks and are the authors of the new book Real-time

0:48Analytics with Apache Spark Structured Streaming. Hi guys. >> Hi Youssef. >> I'm really, really pleased to meet you, but before we dive into this data engineers' journey, let's meet you. Let's get to know you. Who are you and what makes your book different from the typical coding tutorial out there? >> Thank you, Youssef. Hi. I am Subhodip Chandra. I am former

1:10deployed engineer here at Databricks. I am based out of Toronto, Canada. As per my career journey, it started everything with data warehouse and over the past several years, I have worked with uh data platform engineering, data engineering, real-time analytics, data governance, and currently helping our customers to design and build scalable systems and everything with Databricks.

1:33>> Thank you for having us, Youssef. Hi everyone. I'm Harsha Pasala, a specialist solutions architect at Databricks, Canada. And for the past decade, I've been in the heart of data space. I've helped organizations build, design, and scale their data infrastructure and architectures at scale. Coming to the book, why why did we go about this journey? Why is it

1:55unique? The book is a hands-on guide for data practitioners out there on Apache Spark structured streaming. It takes you from raw events to batch processing to all the way until real-time insights and live ML inference at scale. And why why it's different and how it stands out from other tutorials and books out there is it's code-first. You also have access

2:17to a Git repository, and it's also production-first. Throughout the authoring of this book, we've kept kept in mind how organizations are best leveraging projects and how to best deploy them and to meet production latencies. And we also don't stop with an example streaming query or a block of code. We go end-to-end all the way and deploy

2:39that help you deploy that application to production as well. >> Interesting. And in this gen AI era, I think streaming is very important. And what why do you think like data engineers should care about real-time? And once they make the jump, what is the biggest shift required in how they think about data? >> Interesting question. That is exactly the

3:01question that most of our customers ask us. Batch inference, it it answers questions around what happened yesterday or all the way until what happened in the past hour or half hour. But as soon as your job runs, that's where your data feed stops. You process everything that your job has access to until then. You mark your complete marker, you stream insights, and then your job is done.

3:24But beyond that, if you have to answer what happened last minute, what's happening right now, there are several use cases in every industry if you think about it. For example, fraud detection in FinOps, fleet monitoring, live recommendations. None of these can wait for your job to run. None of these can wait for a day, an hour for your job triggers to go and

3:46then start processing. Everything has to be near real-time at the very instant that the event has occurred. The hard part about switching your mindset towards streaming isn't just learning a new API, it's also switching your whole understanding of how to process this new data feeds. Because there's no stop markers anymore. Your data is continuously flowing. This

4:08book walks you through exactly that. How to switch your mindset from batch APIs to structured streaming. And we also cover one of the most recent additions to Spark, structured streaming, that's real-time mode, which helps you get latencies all the way until milliseconds. >> So, just to be sure, once they sit down to write their first streaming job, and as you mentioned, the data isn't single

4:32file anymore because it's flowing through Kafka, Kinesis, and like to flowing through the through the cloud storage, how or what does structured streaming give them, and how can they plug this into sources without it falling apart? >> See, the core idea of structured streaming is very simple. It basically treats a continuous data stream like a

4:55table which is keeps growing. Now, you write almost the same code that you'd write for the batch, and the engine handles the incremental part for you. But, what is really happening under the hood is that the engine provides the two things which are brutally hard to build from the scratch. One is fault tolerance, and another is exactly once

5:17processing guarantees. It manages this through checkpointing. So, if a node suddenly dies, and the stream just picks up exactly where it left off. So, the restartability, that plays a major role here. And the other infrastructure part, like connecting to upstream system like Kafka, Kinesis, those are not that complicated. I would

5:39tell that is like 20% of the complexity. But, the remaining 80% that trips people ups is what happens when things goes wrong. The reprocessing of the data, removal of the duplicate record, durability of the job, so that's where is the real complexity. But, once you get those checkpoints and delivery guarantees configured correctly, the system becomes incredibly robust and

6:03thus it won't fall apart on you. And the bonus point here is that everything is possible to do also in the SQL. So, if you're very comfortable to write your code in SQL, you're not required to learn another programming language. You can write all these streaming queries in plain SQL as well. >> Okay, and Okay, let's suppose that the data starts arriving late and completely

6:27out of order. Something batch processing never forced them course to confront. How do uh this this specificity of windowing aggregation and the watermarking save them here? >> This is an interesting question that every data engineer faces with when they switch from batch to streaming, right? You don't have, as I said, a concept of watermarks or late arriving data. When a

6:49batch processing job runs, your data is just there. You process what's there and then you aggregate your results. But for streaming, as we've discussed, it's just a constantly flowing stream of data and there are uh possibilities of late arriving data. There could be network hindrances. There could be your sensors that went out for a minute. They're not They're not have network connectivity.

7:11As soon as the network connectivity is reestablished, they start to stream all of that data back again. Now, with all of these considerations, uh you'll have to start thinking about how do I deal with this late arriving data. So, you'll have to start thinking from this two perspectives. When did I receive this data, which is the processing time, and also when was this data point actually generated, which is

7:34the event time. So, your sensor could generate at a completely different point. You could end up receiving that particular data point at a whole different point. Now, with both of these in mind, you start to build up on the concepts of watermarks. You'll have to reason about your requirements and you'll have to consider how long should I wait for this later arriving data to arrive. You cannot wait forever, right?

7:57You'll never You'll end up never emitting your results. Also, you cannot just stream instantly. You'll have to go back and correct as soon as you stream because >> If I'm not wrong, if you don't stop the watermark, if you keep like listening, it will will overload also the system. >> Exactly. Exactly. There's Whenever you use watermarks, there's something called

8:19state that Spark employs on your behalf. State uh lives on cluster memory and you exhaust cluster resources if you keep waiting forever. On the flip side, if you if your watermark is closed too soon, then you're not accommodating for all of the scenarios that we discussed, right? So, we'll discuss about the tradeoffs between both of these approaches, how to balance between these

8:42two deliver the results that you're looking for your use cases. Once these concepts click for any data engineer, streaming and the whole concept of real-time processing, it stops feeling so fragile and it feels like this is a controllable concept and you are in control, you're in the front seat. >> Okay. And now that uh let's suppose my pipeline is currently running, I I added

9:05the watermark, added the aggregation to compute the KPI, but the business came back and says, "Well, microbatching isn't isn't fast enough for use cases like you gave previously, fraud detection or live alerts." And also mentioned the beginning real-time mode. So, what's this real-time mode and how does it get them down to the single-digit millisecond latency?

9:28>> Yeah, this is an excellent concept. So, basically, classic structured streaming that runs in microbatches. It collects a little slice of data and then process it and repeat that cycle. And that is great if you are okay with second-level latency, but some workload need to react in millisecond. Think about a detecting

9:50fraud system which where you are swiping your credit card and you need to immediately get an alert whether your transaction is approved or rejected. You cannot just wait for milliseconds for that or triggering a critical alert from a live system. For those use cases, you need to react immediately in fraction of seconds. And that's that's where we need this real-time mode. So, real-time mode

10:14changes the execution model so records flow through continuously instead of waiting to form a microbatch. Here, we are talking about single-digit millisecond latency. And what I like is that you are not required to change any logic in your code. You are mostly changing the trigger type and some cluster setup. So, the ceiling that used to force people to evaluate a different

10:38system or a different architecture, now everything is possible using real-time mode. And you can use real-time mode inside our Databricks platform and you can build your application where it demands millisecond of latencies. And >> Something important, Databricks did open source real-time mode last year. And if I'm not wrong, it's part of Apache Spark

11:004. I think from 4.1. Yes. And okay. I mentioned first that the business complained about the pipeline being a little bit slow. What if the business now wants machine learning prediction inside the stream? Which means not after the fact. So, how would they actually deploy ML inference into a live pipeline

11:22without, of course, breaking it? >> This is an interesting area we are getting into. We are streaming data now in near real time. We have deployed our pipelines for ingestion, but we also have to extract some value out of this, right? And this is also an area where your typical data engineer extracts their realm and ventures into something something beyond what they've experienced before, machine learning

11:44inference. Our book chapter eight specifically walks you through these areas. It walks you through how to train your models using MLflow, which is our prescribed way of training and tracking models, and also how to bring them into your streams, make these predictions inferences in real time within your stream, and serve them wherever you'd

12:06want it to be served. And also finally, this isn't a one-time thing, right? It machine learning, the whole concept of training your models isn't a once and done. Your data and your nature of data, the patterns, they constantly keep evolving, and you also have to constantly monitor your performance of the models so that there could be a point where you you decide, "Hey, this

12:29is model is drifting. I'll have to retrain the model based on the new patterns." We also provide you tools on how to detect model drifting, how to retrain, how to keep your model up to date. All in all, real-time ML inferences, it's not just an add-on that lives outside of structured streaming pipelines. It should be part of your streaming design. It should not be an

12:51afterthought. It should be an intricate part of your streaming design, which is exactly what our book walks you through. >> That's very very interesting, and I think I have pretty hard question. I think all developers develop locally, and let's say now the prototype finally works, of course, but now it has to survive production. So, how am I supposed or how do they handle the last

13:14mile of deployment? Talking here about CICD and monitoring stream that course will run that needs to run for for forever. >> So, you mean the part where most tutorials and webinars give out there. It's the last mile, the declarative automation bundles and Databricks is the prescribed way of deploying your jobs,

13:37configs, bundling your code, streaming pipelines all together. And this plugs in really well with any Git repository systems that you use, for example, GitHub or Azure DevOps. We provide examples for a project that to package is all of these together and deploys across multiple levels of environments. That's the whole idea of

13:59declarative automation bundles. You write your code once and you're able to reuse the same code with some overrides deployed across multiple tiers of environment, dev, QA, prod. You're able also able to, as we've discussed, deploy this within your integrate this within your deployment pipelines. So, all of this gets automatically triggered as soon as your developers commit their

14:21push their commits into your Git repositories. >> And let's say let's look back at this entire journey. So, we talked first about why should I switch to from batch to streaming, understanding some basic concepts like windowing and and the watermark, then talking about the real-time mode, the CI and the integrating streaming pipeline with

14:44machine learning, pushing my stream to course production. But now, tell me a bit more about what does a complete end-to-end real-time system look like once it's built. And of course, this is something important because of your book, where can our viewers grab a copy of it? >> Yeah. >> [snorts] >> This book concludes with an end-to-end

15:07capstone project, a real-time e-commerce clickstream pipeline that predicts the purchase conversion life. It is built entirely on Databricks following medallion architecture taking you from raw data ignition to all the way to a fully monitored and deployed system, a production grade end-to-end pipe. And really, that is the payoff of the whole

15:29journey. So, our engineers started out terrified of batch data that won't sit still and ended up owning up a production grade real-time ML platform. Every chapter tackles exactly one problem on that path in the exact order you'd actually hit them in the real world. So, and if you are working with data and you feel like real-time

15:53processing is coming to your world, this book is exactly for you. It should be in the map. The book is real-time analytics with Apache Spark and it is available in Amazon across the globe. >> And I think one additional more thing, make sure to use Databricks free edition. No credit card are required. You can set up your workspace within

16:15less than 30 seconds and that's where you can apply, you can run your streaming pipelines. You can use Genie code to troubleshoot your pipelines. Also, Genie code to Genie one to ask questions once those KPIs has been aggregated. Thank you so much, guys. It was real pleasure to host you. Thanks again for writing this amazing book and

16:40see you soon.