Unity Catalog Compute: Dedicated vs. Standard Access Modes

0:06hi today I have stepania Leon director of product uh data breaks hi Sania hi youf nice to be here I'm super happy to have her today uh she will uh talk about the UC compute and I know a lot of people have been asking questions about the evolution of the compute since uh we started rolling out UC and stepan is the

0:30right person she will she will tell us the story and how it's evolving and tell us more about the latest features when we introduce Unity catalog we did that because everyone wanted to use uh still wants to use um data governance rules in order to protect an Enterprise data there's a lot of very um sensitive data such as pii data um there's many

0:53collaborators so not everyone should see everything so um at data breaks we need to make sure um governance is always in force especially now that we have Unity catalog but back then um what we did have was Data break sequel uh which had something called table accles where you could Define grants on tables and Views

1:15and so on um but we also wanted this to work for spark workloads in pton and Scholar right and we did have this cluster type High concurrency uh which tried some of the work we did later on um but it ended up with a very very limited API surface area so why was that so basically there were two challenges

1:38if you want to have governance and Spark workloads in pattern and scalar so spark does not really know about governance right so it it was built as a single user single application framework users run as root so it's a shared jvm if you are multiple users H you share the jvm as well with the engine you have privileged machine access so it's not

2:00really an ideal environment to enforce governance where like I shouldn't read your data and you shouldn't read my data and so on um but of course our customers they're used to like sharing compute so you can share you can reduce the cost you have better utilization and also we as data breaks we want to have like a scalable serverless platform right so ideally we can build that on um sharable

2:22compute so the challenge was like how can we like securely share compute and enforce governance so we did a couple of things um which is basically what you see on this slide we started with um what will be called the dedicated access mode was initially called the single user access mode so there we said okay um in order to um make sure um data

2:44governance is always enforced instead of sharing compute we just have a single user on the cluster right so no one can read anyone else's data and so on um there was one problem um we didn't have fine gr Access Control until like 15 for LTS so why was that it's because when spark processes um a view or a table

3:07with RLS CM like row filters column masks um What spark does it like fetches all the tables um and with that like all the underlying data and then does the processing locally so this would mean like I could just if I would want to just go and like read data I may not have been granted access to bre if you have like some roll filters like you

3:31shouldn't read the whole table that's why we have them so um but then what we did in order to circumvent that is um in with dbr 154 we introduced Fang gr Access Control where like all these queries that uh run over views or over tables with RLS CM run on like dedicated background compute and only the results

3:53are stream back to the spark server so that's what you get on dedicated access mode you get the full unrestricted sparkk API um you get full Fang gr access control from 154 onwards um and very recently we also added the ability to share that cluster now you can only share it within a group of users with the same

4:17permission so that's why the cluster is then dedicated to that group so if I'm and if you and I are on the cluster let's say for the podcast collaboration um only data that the podcast collaboration ation group can read can actually be read on that cluster like I can't read my data you can't read your other data just to make sure we're in a kind of secure environment yeah so that's dedicated access mode um but then

4:42um we also wanted to do better is like can we have a secure multi-user cluster uh where we have full data governance and we can run spark workloads in scholar patter and so on so we did that um by kind of taking a step back and looking how we can get there so how can we make sure that on spark we can

5:05isolate user code like that we don't share jvms anymore across users and so on so we did a couple of things one was um we used spark connect spark connect basically introduced a decoupling between the Spark engine and the client application so it goes through the spark connect protocol which is based on the data frame API and it really decouples

5:28the engine um from the client so you don't have like two separate jvms for each client like each client has a jvm or uh the server has its own jbm so you don't have like peny clashes anymore and things like that but also you don't share the jvm anymore so you don't see like the problem we had to use the filtering for f gr a control you don't

5:50have that anymore now because as a user you can't access the engine um but that wasn't enough so because if you and I want to share a cluster I shouldn't read like what you fetched you shouldn't read what I fetched so we also had to isolate our client application like the notbook the job um so that uh those run like fully isolated with their own jbms and

6:15then there's other user codes such as udfs those running on executors they're also fully isolated so that's our standard access mode where you now can run fully isolated um spark workloads in Scola in Python in SQL um the API surface area is based on Spar connect

6:38which is the data frame API or data set API for Scala um and and this is also the foundation for our serus compute so you can can um yeah basically share compute with full Fang Grint access control without any additional cost and just something I wanted to Circle back um if someone is using the uh dedicated

7:01access mode for development please don't switch it to shared or standard and vice versa so if you're using a mode then keep it for D production else you may have a different Behavior because some rpis doesn't work on dedicate access mode and the opposite can be true as well yeah exactly that's very important

7:23like stick to your architecture because the API is different um yeah definitely and and I guess we have also um with the new names we have also new AP like UI API UI for cluster creation yeah exactly so today um on in Databricks if you're opted into the group cluster public

7:46preview uh you can opt in at the workspace level you already get the new ux where you see the dedicated access mode and the standard access mode by default the standard access mode is picked um um this will um become available as an optin um in mid-February so it it comes with a simplification of

8:08the cluster creation ux um so that will be available as an opin in mid February and then will become the default um later in the year so basically then every time you create um a cluster through the UI the default settings would be on standard clusters we're also working on the API but that's um a bit further out perfect thank you thank you

8:30so much Stephania for all those insights