Master Databricks Volumes: Step-by-Step Tutorial

0:01hello my name is Yousef Bernie and today I have the pleasure to introduce volumes in unity catalog you all know that the lake house stores incredibly large amounts of data in arbitrary formats while much of it is governed through tables and many use cases require access to non-tabular data such as working with arbitrary files

0:23including image audio video PDF or XML files to address this database is on reducing values volumes are new type of object that's catalog collection of files in unity catalog and help you build scalable file based applications that read and process large collection of data irrespective of its format including unstructured

0:47semi-structured and structured data this enables you to access store and organize non-fabular data in unity catalog along with the tabular data providing unified Discovery and governance experience the files are accessible through the UI spark API fuse DP utils databricks CLI

1:09and terraform volumes unlock new processing capabilities for the data governed by the unity catalog particularly for machine learning and data science workloads the first thing I will do is creating a catalog and a database and then I have two options I can either create a manage volume or an external

1:33one to create a manage volume I just need to write the SQL query for each value plus the name of the volume and if I want to create an external volume I need to specify the keyword external and location and now that I have created my volume I

1:55can go to data Explorer and of course I can use the Explorer I have over here look for football UK and of course I can display the volume and see I have created the volume data sets go to data Explorer and look for football UK

2:17I will access my database if I go to volume I can create the volume either a manage or an external one through the UI I can handle the different permissions read volume write volume or privileges and I can also upload files

2:40for my case I would bring 10 CSV files but I will upload and if I go to Pros I'll copy the URL of this data and now I'll go back to my uh notebook I can't drop the volume if I want using the drop volume command I can print

3:03permissions using SQL queries I can show grants to see the permissions granted over this specific value I can add comments to help data scene understand the data stored in this specific volume and I can also describe the volume to have additional information like the type whether it's manage external comments and so on

3:27and now what I will do I will create my first Delta table and I will use autoloader to ingest the data I have stored inside volumes this data will be stored in a table called games that will query in a few and after I will transform this data to

3:53add or to create additional colors so if I query this table that's how this table looks like and I will run this command that will transform disable and create an additional one using code games clean and let me show

4:16you how the new tables look look like so I did add additional columns if I go to data Explorer again I will show you the lineage and you will see that you have a pretty interesting information over there so if I go to football UK all seasons

4:41if I click on games and then lineage you will see that they have the volume over here which means I had the com the full lineage that volumes data set was used to create a table called games and then a table called games was used to create a table called games clean and that's not

5:03everything if I go to the path you will see the path of the volume I have uh used and I can also if I refresh you can also browse the different uh folder you have inside your uh your volumes and that's not everything you can also

5:25use volumes to store your artifacts and you can use them of course and read those artifacts or for my case I'm using uh volume to store CSV file and I'm using pandas to read from those CSV files then I'm adding some Transformations and then of course I can use them to train my ml

5:49model thank you for watching this video