Chuck Data: Automate Data Engineering with Natural Language

0:04I have the pleasure to have kebab, chief hacking officer at umperity. I think you have a surprise for us. >> Yes, today we are launching Chuck data. Uh Chuck data is a agent for doing data engineering work in Databricks built natively on Databricks. So chuck runs uh as a python command line tool. You can see uh you know here it it literally

0:26directly interacts with your Databricks tool toolkit. So it Chuck has a library of tools that it can use. You can see that there are catalog tools here um SQL warehouse tools to run SQL modeling. You can see that it has the ability to profile and tag PII data as well as run imper machine learning stitch algorithm. And then you can do all this by natural

0:48language. So the idea here is to instead of writing ETLs, you can interact with a a a natural language with an agent and have it configure and implement things in your Databricks account. So in this example, you can see I just asked it at the top, what customer data do I have? And it looks at my Unity catalog and says, here's what tables are available uh uh you know, and it looks at each of them and profiles them and says, oh

1:11well, these are the different types of PII in there. And you apply those tags to Unity catalog to power governance workflows, that kind of thing. Um and then if you want you can even say hey I want to build an identity graph and so you can see here I said hey run stitch on this data and it came through and constructs a job for me. It gives me an opportunity to give feedback and adjust it and then it actually will launch that

1:33job. It'll say hey I'm going to create this notebook to show you the results. I'm going to create these tables with your links and your your nodes and I'm going to run it with this job and you can open that directly in Databricks. this whole this whole idea is that instead of writing ETLs, you can just use this tool um and and you know it'll operate on your infrastructure in your security bubble. And so you can see if I go to Databricks, it'll actually run that job. You know, you don't have to

1:56understand this this gibberish right now, but um but once that's done, it it has a stitch report that will let you evaluate the resulting data. You can see how those data sets overlap. You can see, you know, duplication rates by different data sets. you can see uh you know how many people went from anonymous to known. So like in this example you can see these were partially known and after the new clusters have more

2:18linked data so there's more people that you can actually target and and build analytics on. Our whole idea here is to flip the concept of ETL on its head and let people interact with a uh an agent that will implement data engineering use cases using Databricks. So um we it's free to use. It's a Python uh uh library that we are, you know, excited to get as much user feedback on as we can. Uh you

2:42know, contributors welcome and and we just want to see how people uh respond to this and as a new way to do data engineering. So, I'm excited to see what people think. >> Thank you so much, Caleb.