Lakeflow & Unity Catalog: DAIS 2024 Announcements

0:04hey today I have a special guest it's my it's our first face tof face meeting I don't think I need to introduce him good afternoon from Paris office so I'm Hubert uh now I'm datab bricks beon so it's super awesome to finally meet

0:27you in person so but first I think you need maybe to introduce the biger program because maybe they're not aware of this program uh so when you are active in Databricks Community uh the most active ones are rewarded as a Databricks

0:51becons uh so then you can join like slack Channel or or make some uh you have access to some inclusive information about uh about datab bricks uh so basically this is

1:14it so it's very like I've seen I think at least 20 FS on the beacons program I will add the uh list of the beacons in the description of the video so Huber tell me dice took place three weeks ago something like this or maybe two or

1:36maybe two tell me more about like the highlights or your favorite announcements yeah I think that my favorite announcement can be surprise for many but my favorite announcement is whole lake flow concept uh because uh t current R is

2:01like to do things easier and faster yes and I like that data RIS uh has finally offering a flow code solution for data ingestion and transformation but what I really like even more is that the slow code under the hood is creating SQL code

2:23yes indeed yes so also also for years we had this discussion do you need to use qu or python for uh EPL it was like never ending discussion yes in community and my opinion that discussion didn't make much sense because uh p park is like API to spark SQL uh but of course

2:48in Python you have this meta programming but in my opinion thanks to the uh L flow concept it's like putting answer to that question what you have to use for ETL because basically the code which it will generate it will be SQL code yes so uh so I I really like it and I really

3:11like whole concept of implementing their best practices so concentrating on change data feed have like ready native connectors to many sources so it will make your life really easier because what I see that companies are strling to managing their connector or different

3:34rest apis to a lot of sources like like even this SAS platforms like for example Facebook connector for marketing team or something everything could be native yes because when it's native someone is responsible for that connector yes and that's why I really of course the

3:56most important news was Unity catalog which is now open uh open source but what I really like with open source in community catalog that it will be also faster to deliver like for example Uh custom connectors yes because this way uh all my vision is that all companies

4:18in the world have now access to Unity catalog code and soon all of them can start uh start developing Unity catalog for example to bring their s platforms or under Lakehouse Federation yes uh so in my opinion it's really really awesome to create all this environment combining open source sare

4:43low code so it's really awesome and and also the another news also like also in my opinion a bit like uh squl and uh low code is that uh that whole idea that you can put uh custom function into llm models yes so

5:08yeah I was always fun everybody knows that about registering function in unity catalog that your code is visible to business and now you can use that function like buy them to some events in llm like someone I don't know you have compound llm like internal and someone say that I don't know this water is not

5:30working on third floor and it's like uh have this act function which is trigger when there is a problem open J ticket yes because in that uh Unity catalog function you can put also python code so it can make rest API uh call to jira for example so that was really amazing that was a demo that uh uh Casey showcased

5:55she was calling functions and she built some function that that she was calling in the playground to and those playground like those functions were calling the um the new shatter stock image so to build to create this cookie and send the message on on slack but just to go back to L flow to clarify so Lake flow we have three uh let's say

6:20components we have l l flow connector so that's what uh hubber was talking about so we have native and scalable connectors uh I think first going to start with um fromr Salesforce and workday and we have a bunch of other connectors coming later and then we have the second part which is L flow pipelines uh this is where we're going

6:43to write syntax to transform your transform your code and then you have the last part which is leg flow jobs so this is the orchestration uh part and behind SC going to handle the as you mentioned CDC the um incremental inje handling the schema Evolution the failures and everything is handled governed by unity

7:06catalog and have this observability to track the uh what went wrong what went well and the goal is make the life of the data Engineers easy and you have this of course this this end to end lineage which is I think very also very very uh important so for me that was a pretty

7:28pretty cool announcement and also mention Unity catalog so yeah I'm proud of my t-shirt thank you thanks Denny and Michelle for for the shirt I think it's very also important because now you can use other engine like d DB to write into Unity catalog which is super cool which means you can build your own applications using ddb

7:52tro or other andin and write and read from from UC and this is super super cool and it's already available I think this is this is something I need to mention it's not in few weeks but there you can just go to the repo I will I will put it in the uh in the link description of the video and you have also our I think they had yesterday their first uh Community uh meeting so

8:19make sure to join yeah I saw I saw that d d made it yes yeah exactly exactly yeah yeah so uh so also big announcement was uh like integration of Delta and Iceberg yes so so I I hope that the that both formats will be like unified that

8:42that the one format something like Del Iceberg will will come from that yes so I I'm really happy with that announcement too so acquisition yes so I think we had a lot of interesting announcements yes and I think they will be keep coming because for example for Native connectors there is some someone

9:05counted that there is like, 500 possible native connectors which could be built yet because there is so so much platform and databases on the market so there is a lot of uh work above us yes and uh uh and also I think in we in next years we will have

9:29uh like I think more more boring announcements are coming because uh we as a like data Engineers we will have to work more on the things like governance and security managing private data so so in my opinion we have really like the best platform in the world to manage it data breaks and now we need to add to

9:53that platform all this small small and boring like stuff everything what this meing yes so sometimes there are like small announcements like there was this now option added budget yes that you can specify B yeah so so so there are a lot of the small like small ones which makes this platform like the perfect one yes

10:15because everybody of us know what what is missing yes and we can add all this uh all the pieces and I think Databricks is like uh the best place because of this mixing of like Enterprise Style with Community yes that we can we can achieve really a lot yes so all

10:38because what was frustrating me like 10 years ago when I was working on a lot of different platforms was always this lack of Engagement that there was no such a community and for example you say that you you put in some backlog that you would like to have some feature and then you waited 5 years uh like often with

11:00Microsoft products yes and after five years the nothing changed yes it was still in the back look I don't know to export some report or something yes and in the briak is that I'm really Amis because you say that I don't know we need such a such a function I I remember that like I requested some small function some

11:24some small improvements sometimes it was like I don't know how how you click uh run button on the Notebook and all these things and all was done so all improvements are done really fast yes thanks to data Bri compan and thanks to thanks to community which can always contribute and uh and make make it

11:48really like the variant type yes you can say this is like from spark like Comm done by community and data something which will solve life a lot of problems and that that that's what I was about to say what I like about data breaks that the platform is built by the community for the community and this is

12:12I think this is something very very important and you mentioned just fact of delivering fast so I've seen the notebooks like just going to take small components of the platform notebooks has been evolving so fast like you've SE now have the possibility to do the buing inside the not books you can uh like do dat Exploration with just

12:34some your with doing some Dragon drop they're adding the uh the um how do you call it uh the um the markdown like better UI simplifying like your the dev experience and this is only a small component like and all the others like workflows has also been like advancing so fast they're adding they just

12:57announced like the looping can now loop on the workflow you can have triggers with data table so because I've seen some customers struggling how should I uh when should I trigger my job because sometimes the data does not arrive I don't know at 10: a.m. and now they can trigger this job when the data is updated on a specific one or many

13:20many other tables so I've seen the all the different products evolving so fast and of course varant this is also highlight of the data Summit it's going to be part of it's already available in I think the DPR 15.2 and it's spark of part of spark four so variant type is easy way to uh

13:45work with Jason so before we needed to uh to par this seon to string to make sure that you uh try to access this data easily but it was a bit bit slow so we had the two option either infer the schema which is not always good or declare the SCH the schema which is not

14:07always flexible so with this variant type it's uh it's there is an Implement implementation of a bit map how you can track the different rows and you can just use the function parore Jon add your type your your your G and then you can access it easily and access all the different different columns and it's up

14:30to 10 times faster than the actual method you you're using and variant is 100% op the code is already available in GitHub yeah for me the most important is that when you get Json and the schema is changing you can you can just use it save it as variant type and the fields will be in the uh in the correct uh

14:53format yes like it's some part of Json can be integer some string and then next time it disappearing from that Json so it can be all handled so it was uh for developers and data Engineers this handling Json was a big big headache yes and now it's solved so I think like information for everyone is if you have

15:16any like constant problem with Databricks and you know that sing should be improved it can be done yes and it's uh it's really amazing yeah and also talked about the uh Iceberg so I don't know if you I've seen the announcement but as part of Del 4.0 Delta uniform is now

15:40GA which is good which means you can use Delta table and generates metadata for Iceberg and Hoodie tables as well have also liquid clustering and I think also this is something uh I've been Amazed by this by this feure I don't know if you use it that much but it's giving me more flexibility now I can change the logical

16:03partitioning in very easy way rather than having to rewrite the table every time I need to change this so it's giving me more flexibility also on SQL how you possible to set liquid clustering by Auto So based on with time based on the read and right patterns it will decide what What fields use to partitioning yes so so

16:28basically thanks to all this Improvement especially on SQL Warehouse the speed of Databricks is much more faster for this like traditional SQL Warehouse usage yes because for example this uh this like like combining AI with the SQL Warehouse because it's like governed by some model yes to decide for this like

16:53index less indexes yes because this way it can read read and write patterns and with time it will decide what liquid partitioning is the best for your table so so in fact it's solving like problem with indexes yes because of course on on Delta Lake we don't have like uh real

17:16indexes yes but thanks to that liquid partitioning for in my opinion it's solving many many of these problems but probably more more needs to be added but I'm really amazed how the speed of working on SQL Warehouse was was improved yes so basically when you when you ask some like questions it's you get

17:39answers really really fast yes so so it's like like I I cannot like compare what was two years ago and now yes it's like yeah yeah now now it's like I think rolds mentioned this like in his presentation like he was showcasing how the performance were improved since we

18:02launched data break SQL I think three years ago and it show show showing that it's really like going like very very fast and also one of the fastest uh adopted product I think in in data braks and it keeps growing and I think also this is something Franco patano keeps saying data warehousing is a solved problem

18:25yeah or the S and and also like when I keep thinking like we introduce predictive optimization like this feature that help you optimize automatically the tables and run the vacuuming and then we did introduce liquid clustering at the same time we are with this predict optimization we're going to start capturing statistics like

18:49right instead of having to run the analyze to capture statistics for various columns it will be done automatically and based on the query pattern they will be able to choose the clustering uh clustering key like as you mentioned cluster by Auto which means you're going to see like all those features are link because you uh optimize the table automatically you

19:11capture the statistics and the query pattern and you're able to get this which means there is a flow and like when you start building those products I mean it's pretty hard for me to imagine like they they have this big picture of what they're trying to achieve and this is something I really love about data bre there is this uh they always like one step ahead like I'm talking about

19:33being a brickster but it's I'm always amazed by this kind of like how they build the structures and the different steps to reach a goal and and I'm really Keen to see what will be unveiled for next year or for the next two years yeah of course so I think the answer is that we can do everything now faster

19:56better and cheaper yes because because basically thanks to all this improvements you don't need to spend time on thinking okay I need to add this analyze comment to my table and run it yes in in fact uh Mo most people most project uh haven't used it any way yes that know I haven't seen more much much

20:21analyze syntax in in production yes so uh so in fact thanks to all this Improvement is like out out of the box yes and I think this this is a strategy that data breaks is offering like ready ready to use Solutions yes and reduce the overhead and just let you focus on what's bringing the value for your

20:46business and rather there need to handle all these uh various workloads or various tasks so yeah that but of course if you want to keep doing this on the classical oldfashioned way you can still doing it if you want to upload those TS to this to the platform it can be done so you have this uh this flexibility yeah

21:08also uh also there was this announcement about serverless and here I want to know your your opinion yes so so that everything can be run on serverless including like notebooks we can run notebooks now on serverless but a time like it's easy use but because there is one version of serverless yes

21:32and and for for example for me it's problem yes because uh when you do some for example machine learning in Python so not native Spar code there can be a lot of changes depends on version so in fact I would prefer to have like few version of serverless yes to uh to know

21:55what is the uh what is what what is uh libraries behind it what is your opinion on that yeah so this is also something we thought about it so I will split this into into in two parts so the first part so far now you cannot uh run ML on the serverless you cannot do much you

22:18can do we have some some Frameworks that are already support like pyit learn but if you go back a bit in the like a little bit further so servus architecture is based on shared clusters yeah so this is a new architecture there is two types there is single mode single user and shared one so shared one uh you

22:41will see that you you can't find in any ml runtime now so you will still need to use a single user but for you can run psyit learn on the shared cluster which means you cannot for now use serverless to train your model or stuff like this so but it will change eventually but yeah because there was this issue

23:03recently that uh python version was upgraded and after that Panda stopped working so so it's always for me like when you have SQL it's easiest because you can control syntax it's SQL syntax but you when you have a lot of external Liber It's tricky to sometimes to go

23:25serverless yes so like with one version for the second part you mention about libraries so there is a new uh feature that will be added to server L actually already tested in private preview we're going to have a variable uh let's say or it's like an environment part where we can set up your libraries on your own

23:47like and you can okay it's not installed one and actually it's going to change they're going to decouple the uh the libraries from the from the uh from the compute so from serverless so it could be great because then you just have serverless where is just Python and and Spark yes and then you can add you're

24:09going to have some libraries by default and you can add the others why because you know that now when you uh start a cluster there is a startup time but there is also some time you spend to set up all the libraries and sometimes don't need 20,000 libraries just need five or six so this is something they're gonna bring like add in EnV

24:32like a new UI called environment if I'm not wrong where you can add those spe specific libraries and there is a specific talk about it AD DI for those who has access to the breakout sessions there is specific talks about server data break Ser offering and how this feature will be helpful to handle those kind of uh issues and also what is

24:57your like opinion about spark connect because now you don't need to use p p spark anymore because it's spark connect so you can use go or or r or some other languages yes so is your opinion that someday we'll go to Databricks and you will have notebooks in like all possible languages or or rather not I think I

25:21think yes I mean they they did first of all like for those who don't know what Spar connect were they coupling the uh client from the server which is very helpful and when you first introduce this spark connect like you can just I don't know connect your uh you're using your LTAP you're going to run the code but your code will will use the data

25:44briak computer for example and this is something this is something cool because we going to you you decouple this but of course also offer more possibilities like uh as mention like you have this Ras or you can develop with with other languages and I'm really ke to see what the community will bring because as you as you know aach spark is open source so

26:08the community is contributing massively and let's see where this will I think will lead us yeah go or R is already for available for spark connect yes so I think Russ is evolving very very fast the r like the r Community yeah because it's faster than Pon yes so sometimes

26:30it's important for some machine learning stuff and I think they also started working on the uh Unity catalog rest connector if I'm not wrong so they just created the repo but this is something they are Keen to work to work on so feel free to join the discussions to discuss with the people behind the this this

26:54report and what about jie a AI B I always don't remember is this AI bi or B ai ai B yeah I think it's time to for data breaks to enter bi like more seriously yes also to be honest honestly Microsoft

27:20is like monopolis with powerbi and now they are changing Li license terms to make it more expensive so I really would like to have datab bricks uh bi app which I install on my desktop which can be even embedded version of datab bricks dashboards and then I can show it to business that is datab bricks bi and you

27:43can make bi with Databricks you don't need to buy other tools yes yeah I I I really would love yeah that approach like the think as aibi dashboards or Genie was were introduced we are not here like to replace uh forbi

28:05Tableau yeah other tool I think the goal here was more to uh offer a platform where they can use AI to have insights quickly like you're I don't know you're CTO or your CIO you want to just have an idea how what what was like your I don't know sales last year here or I don't

28:28know comparison between those you just need to go and ask the question Revenue uh by product for the last 10 years and you have the you have this nice UI nice chart displayed in front of you and like instantly and this is cool because you don't need to rely on data analyst to build this to build this query for you because you just thought about it and

28:51you want to add it to specific slide or just for a manager to understand what's wrong with the business and how they can solve the solve the problem I think the goal is was the goal with aibi was more to democratize access to data and AI to non-technical folks and this is I think

29:13this is where we are leaning to leaning to go with this text to text to sequel yeah but but non technical FKS always come to you and say create for us a dashboard using Self Service software so so in practice it's working like a bit a bit differently my

29:35in my opinion yes if we have apis with Genie for example you can maybe I don't know build an application on I don't know datab braks apps this is something we introduced one year ago called Lakehouse apps now it has been rebranded to data breakes app it's like you can add the stream stream L application or any or grad application where for

29:57example you can just use the API to ask question and then it send I don't know a graph to show the results like can you can add it I don't know to we pay web application or this like this datab breas app so we have options the road map has been is currently now shaped and I'm really I'm Keen to see how this AI gen

30:20will uh will evolve but so far it's G I tried it three months ago and I was amazed you know what I did I went to the um I downloaded an open data set and I just upload like I added I upload this manually to data brakes created the table and I generated the

30:42comments automatically with the AI generates and then I opened like Genie and start asking question said okay please show me all the Chargers available in uh of H which is the let's say the region in in France and it did understand automatically that when it's in the

31:04France those are the postal code and it's filtered automatically with all the available postal codes so I I don't know them by heart but the Gen generate this automatically for me and give me the result say okay on those three like on the results I get like aggregate them buy them by uh by companies and filter out the companies that doesn't exist

31:28like like not doesn't exist but the null values and automatically excluded the null value the null values and it like showed me the result and it was like so fast and that was good I mean I was really really amazed and of course more insight you add with this information on the metadata like more accurate the

31:51result like problem with AI andm was that it was B not to like as a large language model not the model to see the data yes a whole table inside so I think the next steps will be that integrating more llms model with not metadata but with real data yes because it could use

32:15like samples or something like this some some even in in parets in some files you have this min max values yes the headers yes and uh and it this uh J I could read it yes because because when you have good metadata and field description is easy but sometimes I I see in many like

32:38clients that they have like this the name of the field which no one understand what is it yes like x z yes or something like this okay because it's some company internal symbol code yes and there is no comment with thep encrypted column yeah yeah yeah yeah so so sometimes it's uh it's difficult but

33:00but it's uh it's evolving and in my opinion Banks like to feedback from customers it can it can evolve and be uh be better yes and then it's executing that queries of course on SQL Warehouse so all this Improvement to SQL Warehouse are also important for this gen J B AI

33:24part yes because it's utilizing SQL house under the hood so it's utilizing the cashing and all this advantages yeah um I think last question would be about uh what do you think about the attribute based access controls on UC yeah I know that is your favorite topic yes but like I think we need it because

33:49we need more ways to govern security because companies are really lost with all this groups inside other groups which because select here use schema here use catalog here execute here permission so maybe the like tax system like attribute bases is more like

34:11intuitive yes to call that to make like T finance and the finance you you see to what uh what what has access to yes so so there is a lot of to do in the in that part is to make it easier and even that can be connected to some AI system because for me it's really headaches always to analyze anything related to

34:35ACL yes to governance of access rights yes there are so many tables and this like environment is growing yes because you have you have some access rights everywhere like experiments uh Vector search pipe data life pipeline Vector search end points everything is some ACL yes so it could

34:58be it would be really nice to like to have for example this attributes and one more question what would you expect for next year for dice for the DAT na Summit announcements next next year or what would be your like what what do you expect or if you have any if

35:22you have one wish what what this uh what would be yeah uh for for me one one wish is to have like really a lot of native connectors but it's like long long road yes uh because I always struggling with that like my entire life that I need to connect something to something yes and

35:45because it's always the first step of at the project yes bringing the data yeah to make this connection yes so when I have more native it's it will make more my life easier and then it's easier for me to propose to someone who Databricks yes because he say I we have this type of system or servers yes I say okay data

36:08bricks have connectors to it yes so so this is and and also like I I I heard what you told me about bi like that that is like more about democratizing data but I really generally to be honest like I hate all the

36:30tools and and I really really like would like to have like I don't know the that I can make this few charts like really simple charts always in Databricks and in fact one only thing which I'm missing is there is to have tabs on the dashboard so so it would be awesome for

36:52me like to for example to have tabs on the dashboard yes so this is something also uh if I'm not wrong Miranda mentioned this but this is something on the road map I know it's on road map because I requested okay so yeah yeah this I think that confirms what I mentioned that yeah it's built by the community for the community yeah

37:15exactly exactly thank you thanks HT thank you for your time thank you