Databricks Workflows: Table Triggers, for-Each & Serverless

0:04hey today uh I'm with hobbert hi hello Yousef good to see you again likewise likewise it has been a while since our last chat yeah last chat was personally in Paris data and word tour and today we have hbert because we want to talk about the latest features uh that has been Unleashed in uh for

0:28Databricks workflows yes there are quite few uh exciting announcement workflows are getting more matu and we can build much more now in Databricks and I expect in fact that this year will be really interesting because we are expecting a lot of new addition to

0:52workflows some were already announced but are not yet available but soon they will be and we can talk about it and and also show some demo about workflows and what are the uh the feature that you're using a lot in workflows yeah I I think Mo uh most people are using the basic features because this is how they are

1:16designed uh but I I found that something uh what is really important for people is how the workforce are triggered that's why addition of Fire arrival trigger was big Improvement because you can have for example volume with files uh where the files are coming

1:38and then autoloader is automatically triggered once fire arrived and it's really awesome the new trigger was announced and it will be available just in few weeks for everyone it's table trigger so it's also really amazing that it's monitoring last update of uh of manage tables so it's also really nice and I think this is like the first step

2:01for everyone is how to trigger the workflow because it's something which is starting starting the flow yeah yeah exactly the different triggers are very very cool like especially the one you mentioned like up like triggering workflow base on the updates of one or or several tables uh okay so we can we can maybe

2:24show workflows I can show the screen yeah yeah okay and then maybe we can talk about some best practices like how about maybe putting some notification alerts some timeouts some retries yeah uh yeah but but first uh I would like to talk when you go to workflow page uh there was some small changes uh

2:47regarding naming convention because now when we click create we have three different type uh uh type of jobs so we have regular jobs but also so from create we can create it l pipeline which is new version of the LT pipeline because uh uh this year it will be huge

3:10progress with DT and this concept and there is also ingestion pipeline so ingestion pipeline uh is also something really amazing and everyone can now see safe Force connector but there are few more in preview and some few more not available through I there is also workday service now and SQL server and

3:36what's amazing with this connectors is that they are built based on uh change data feed and slowly changing Dimension so they are update the tables in destination incrementally out of the box yes so I think it it is the first best practice uh that if something is available in Databricks like data

3:59bricks Conor use it and then if you cannot use it because of course there are hundreds data sources then go to another solution like Azor data factor or pan but this is the first place uh to look for connector yes because it will be not only easiest to implement but also cheapest and the most effective architecture because you consume a

4:22change data Fe yeah and you can to also have later the uh lineage which means you know that you're ingesting data from Salesforce course is pushing it to some specific tables and then you take those tables to transform them and then um using dashboard so you can have this lineage end to end from The Source till the let's say the destination which can be a dashboard maybe aibi dashboards

4:46yeah yeah in fact yes I haven't thought about it but yes it's really great because then the table which is in s force will be included in lineage in Databricks yes automatically okay so let's click create and probably every everyone knows that that page uh regarding for example best practices I

5:09would like to point and new features to few places first is compute but now we can run job on serverless and I did some experimentation and in fact cost in dpu cost of serverless was usually the same as cost of non surveillance but there is no additional costs yes so when you run serverless you don't pay for virtual

5:32machine ITC so it's really it's really nice to run a job on serls and then compare in system tables uh how how your B is changing so it's really nice and also cop optimized option is coming so it will be even cheaper to run on serverless so it's really amazing of course for SQL job you can run SQL

5:54Warehouse so so it's even more better and better optimized and cheaper uh but this is like the first part always to think about compute which which can be cheaper and better and faster uh and another uh place which uh I'm which is really important is to

6:17configure timeout what I found that often people forget about timeout is so easy to set something so you will not finish with uh a zombie job like the python script which is running forever uh from some library and recently also some super option for streaming timeout were added here and

6:41it's really uh really useful because uh because for example when there's no new records on stream you can time out it automatically for example after 20 minutes so uh it's really good to handle it or to even to restart jobs stream job Bas on this time out because maybe something went wrong so I found it's

7:04really useful and another another thing is triggers which we mention that you have FAL trigger it's something which I'm using really a lot of and table trigger is coming maybe if you can tell again a bit more about table trigger do you think that it will be like new best practice to use table triggers Yeah so

7:26basically for table triggers uh let's suppose you have a work worklow that relies on four or five tables so basically in the past you had to specify for example that this this workflow should start at maybe 10:00 a.m. but what if the data doesn't arrive at this specific time slot which means you're going to just run your workflows without any purpose so the goal here is to make

7:49sure that you can trigger the workflow only when the data is being updated on one or several tables and then you're going to ask a question like here but what H what happens if I keep updating this table of course you can have like an there is an advanced configuration where you can specify okay if I run this workflow I should wait maybe 1 hour or

8:11two hours or even more to trigger it again and that's how you can you can be sure that if you're onun your workflow you're going to be taking advantage of it and you're bringing the a to your to your pipe to your company yeah also I'm really excited about table triggers uh what's more it's always good to

8:33implement notif yes also I think the r rise can be good for some specific tasks okay I I think I lost you for a few seconds so continue I said retries can be very very important for some specific task maybe the first time it didn't work so you maybe give it a second shot or maybe third shot before failing maybe the pipeline yeah yeah of course yes so

8:57yeah but but also be sometimes we careful how how many retries so you need to think always about this Ric about timeouts retries and this kind of stuff and uh what more it's always good to implement some notifications uh I can I can say that I'm not using email notification because

9:20uh then you can finish with big garbage in your uh in your email I I really like pagor Duty that I encourage everyone to try and you can install on your mobile and set some schedule with your colleagues who can see the critical Works failing yes but of course do it only for production yes so uh but it's

9:44really really important option and it's nice that P duty is integrated out the box and uh what is more the edit is integrated so it's the same code as in data Bri asset bundles so it's also really nice solution usually I use it just for copy paste to

10:07to code of the bundles yes but but is really nice that it's both both ways are like integrated and I think also the conditional execution like maybe in the type of tasks yes yes yes of course so in in type of task you have all this like logic features now like like if

10:29else so for example I using logic for if else if for example I have incremental streaming uh update of the table uh but if it's fed then I trigger full reload of the table yes to delete shape point and of course it depends how B is the table yes but but you can

10:52implement this kind kind of logic with else to cat that something failed and try to repair it uh also for each is really useful because you can Loop the r uh and execute for example object uh I use it for example to Loop the list of restaurants and then import data from every Resturant of

11:16some client yes so it's it's proven in production that it's working and of course we have this things uh like you can you can run clean room notebook this new addition so uh also clean rooms are like I think datab bricks clean rooms are like the perfect clean rooms yes uh to

11:40collaborate someone but it's just small part of workflow I think what is important for for everyone is just this basic things like Els and uh and this trigger so I think maybe us if you want to collaborate more on some additional features about which I forgot I think my favorite one is for each because in the

12:03past you had to do like some some tricky things now just you can pass parameters of like a list and then this those parameters will be uh like captured by the task and then like trigger the task for each par for each parameter and they are done like uh at the same time which is super super cool which means for example if you have a pipeline that's capturing I don't know returning list of

12:26countries and then this list of country will be parameter for each job to for each task for example to calculate the metrics for France for Argentina for Italy Spain uh sales for each maybe country and of course we have uh the parameters that you have mentioned you can give you can also pass parameters between different tasks uh I think those are those are

12:49very very cool uh features that are that are coming and if I'm not wrong I think used to pass parameters only when when it was like uh python or Scala now you have you will have the possibility to pass parameters to SQL uh tasks as well this something I think coming very coming very soon yeah because there is

13:11this uh markers yes for parameters so it's even it's even nicer in than in Python yes so but uh what I think is really important now I remember is that I found really often that people in Notebook put reference to some like undocumented features like there is this entry point uh to get for example workspace ID or notebook ID or run ID

13:36but the best practice which is really important is that when you clicked here insert Dynamic value you can take here the dynamic value uh for for example these things like notebook P run ID Etc and then you can consume in your script so and it's the same for server for

13:58every around time uh so so you don't create code and pass this way so if you need to know the path if you need to know run ID what I I see is really popular that people use some script datab brick SDK Insight or entry point to get run ID and they loog run ID in

14:20some log table but you can pass it from here yes uh and it's working really nice so I recommend all the metadata pass here from the workflow yeah all all this stuff yeah and I think also one of the things I were added recently is the you can track the different runs with the with sequential like with the new UI to

14:43track the execution of the different tasks and of course the existing one which is repair reruns and stuff which is can be very very interesting I think yes yes and the same with the pipeline that you can now for example refresh single file task yes so it was it was improved uh uh a bit so that's nice and I I think the third pipeline which we

15:05can talk about is uh ETL pipeline it's like new name for DT Pipeline and I I think I will not show so much here because I don't have read the DT pipeline but what I found really amazing recently that materialized views are getting better and better because uh

15:28it's really simple to create materialized view and you can run using pipeline or SQL warehouse and now more and more this materialized view are updated through pipeline incremental I think now it's around six more than 60 percentage and it was like 30 percentage so it's really nice to combine materialized view with your uh Pipeline

15:53and if it's simple SQL then uh then it will be also updated incrementally so I think we can say that there are workflows and there are all these things which are like depend workflows yes so different objects so the work is like the H orchestrate like everything so it's nice that there are Improvement

16:16on both sides yes yeah indeed and and there I think there is also something coming uh soon and this is something also I think best one of the best practices make sure that you tag your pipelines and your job this is very very important in order to make sure that you can uh track the cost and with serverless you're going to have uh something called budget policies um as for example if hbert is

16:41part of the group data engineers then once hbert creates this workflow it will automatically be tagged like business unit data engineering and then at the end for example of the day of the month you can go to system tables with the dashboards you can have built on the top of it you can track the cost of this specific job which can be a very good way to see what are the most expensive

17:03jobs maybe it is worth having a look at them optimize them or maybe doing some I don't know uh I think it's still going to be uh very interesting to use compared to uh clusters where each time you need to tailor them to your specific needs and sometimes you're going to have maybe one terabyte to inest the day after maybe you can have just 20 gab of

17:26data in the past you have to use it it's like number of workers put maybe for example between 1 to 100 choose whether it's memory optimized F optimize a lot of a lot of things to do live with serverless there is no Noob just click on serverless and data BRS will choose what is the per perfect compute resource that needs to run your your uh your

17:50workflow and with what hbert mentioned the cost optimize going to have this possibility to run it at a lower cost but of course you want you want to be having the SLA that you have in the uh um as fast as possible or let's say the uh nor the mode where it executes as long when you hit like execute then the

18:12computer is assigned to you with cost optimize you can have some uh some uh let's say some time before having this resource allocated to you but of course it can be it will be cheaper cheaper than yeah that can be delayed like after 1 hour because it's optimize but for example when you run your workr at the night it's perfect yes because it's yeah

18:35night batches can be very good option yeah yeah yeah yeah then for sure it will be like the best best score but also regarding that tax it's important to remember that for example to train machine claring models or some different use cases people still use standard compute a classic compute non serverless so then the uh like of the tax is that

19:00they are propagated down so if you have your data from Azure even some networking data storage data the tax will be included in oper operations there in detailed building yes so you can you can get information from there as well because even on server L you need to remember that you still will

19:23have some not VM costs or drive cost but you will still still have some some Network cost you will still have course normal storage cost so that's why it's also important because then you can use the St on both ends yes because they are propagated down yeah that's exact I think if you go to maybe the have you

19:45have you run any workflows over here because just want to show that maybe the runs different there was one but it was run quite long ago that's still good for example you can see here the different tax you can see the uh the parameters the logs you can of course if you click on like on the Arrow you can maybe um in case of failure you can find like you can repair

20:09this task and restart from the we yeah you can see on the top edit uh repair we run and it what's good with it is that in case for example you're running a workflow for like that runs I don't know for 3 hours in case a last task fail you don't need when you repair this task don't need to rerun everything from scratch but you will just run from the

20:32from the task uh failed failed one and then you will be saving also some compute time which is very very important of course and actually that job here which has only one task is quite interesting because I developed small data brick app for business user where business user go not to data brick just see that Databricks apps and

20:54select this uh uh the this three parameters in nice form so this faram one to three in nice form and then that uh data pick apps is triggering that job and it's really amazing because uh you can pass you don't even have to pass ID to Databricks apps of your job you can just pass the name and use uh system

21:18variables there for your jobs the all this integration in datab which is the really a lot of them are amazing yes because you can for example uh data up for business user and they just will have play button to run a pipeline and then you can trigger uh browser notification to notify user that your

21:42data were reloaded and it's ready yes so uh this is like I really like data break that it's like complete complete data platform yes yeah indeed and it's making life very easy for for developers yes so uh yes said do you have any else to add

22:05on the workflow part I don't think so for now yes so or maybe I forgot about something yes but but really for most data Engineers is like their like their home of their daily job with Databricks so I think what is important in data breaks that when we have feedback that

22:28something is not working well or or should be improved that feedback is like is really working yes that uh then people in Databricks developers like are uh improving based on people feedback so it's really nice what I like Databricks that things like workflows

22:50are uh are built uh based like on you and if you have any suggestion what could be improved just write on community or write on LinkedIn and we are we are reading it and then we are like proposing to product team to improve that yeah exactly yeah that's

23:12what what you mentioned before like database has been is built by the customer for the customer yeah okay uh I think I will stop sharing because this is the end of my small demo yeah but that was good and I think we have a lot of app Improvement that we cannot talk about right now but maybe we should wait for the right moment

23:35we can talk for now is this ingestion uh pipeline that probably soon that page ingestion will be like busy with logos yes so there'll be more and more and it was public of course which are coming first yes so it's it's really amazing and uh the same like regarding inje on your pipeline now

23:58about sap Databricks is amazing that you can Delta share data from sap to Databricks and then process it further with uh with pipelines and I think for example with materialized views it's like really awesome to process sap data so so I think uh really really

24:20amazing I think it was like news of the year that there is sap Databricks maybe it's not like in the topic of workflow is like really big game changer and I think what is in data briak the most important that there is like really Vision how how it should work and how the complete data platform uh will look yeah indeed and as

24:44you mention going to have more and more logos there in the L Flow Part uh they started with the SQL server but they going to have more data basis and also more SAS I think now uh think work day we have workday Salesforce service now maybe and I think they going to have more and more services coming to leg flow which will simplify the ingestion

25:08and then just uh go and start running your transformation orchestrate everything with the with workflows or orchestrate like flow or yeah I I also like this approach that data brick started with that systems used by big Enterprises like Salesforce and workday uh probably many people from some

25:31companies even don't know what is work day but a lot of big companies really use it the same with service now uh so it's really nice to uh to integrate that for example for service now I see a lot of use cases that you can manage with service now really nice for example a group active directory groups ENT ID you can for example using that connector you

25:54can import that information and you can for example then validate in workflow is uh the security things yeah because components use service now to like to request access to resources but I think is discussion uh for different hour meeting yeah um thank you thank you so much Herber and uh see you in the next

26:17episode yeah yeah yeah I think we will we will talk soon again yeah thank you bye us uh don't leave