Extract and Structure Enterprise Documents: AI Functions on Databricks

0:04Hey folks, I'm back and we have a new guest, Archika Dogra. She's a PM of AI functions at Databricks, but I mean let her uh let her introduce herself. Hi Archika. >> Hey everyone, super excited to be here. Uh thanks for having me, Yousef. And yeah, as you mentioned, I'm a product manager here at Databricks. Um our team

0:26works on all of our functions, which are basically SQL native functions that help our customers turn their unstructured data, whether it's documents, customer reviews, call transcripts, into structured insights. So, super excited to talk about what we've been building over the past few months, um and a lot of our recent releases that have made the story even better than it is um than it was yesterday. >> And I remember um I used the first AI

0:50functions maybe 1 year and a half ago. Uh back then, I released a video about how to get started with those functions. I remember AI analyze sentiment, AI classify, AI grammar, AI translate. I don't remember all of them, but like think list was like nine or 10. What are the new differences between the this older release and the revamped

1:14version? >> Yeah, yeah, no, that's a great question. So, a lot has changed probably since you tried them a year and a half ago. One of the main things is that we actually released a brand new AI function called AI parse document, which essentially helps you turn your messy, unstructured enterprise documents into structured layout and context-aware text. So, it

1:36parses all of the text, images, tables out of that document and kind of saves it as a high-fidelity structured output that then lets you plug that into downstream like agents, analytics, flows, whatever it might be, so that your agents understand your documents even better. Because they really struggle when like you throw an LLM at a raw document to actually read it, parse it, and make sure that it's extracting

1:59the right information from it. So, that's one of the new AI functions that we've released. And then, um we also have revamped two of our core AI functions, AI extract and AI classify, uh which are also super key to like unstructured or intelligent document uh processing. And so, AI extract, some of the main changes there and for classify is that we have introduced a new

2:21interface, so a new V2, that gives customers a lot more agency and flexibility in terms of how they define their schema, and then also what kind of guidance they can give our system behind these functions to make sure that we accomplish the task as accurately as possible. Uh but also, not just like the revamped schema, we have actually thrown a lot of research effort and engineering effort to optimize these functions for

2:44these tasks. So, basically, some of the key improvements that you'll see is around two to three times faster throughput on these core functions. So, your pipelines that you're orchestrating that are using these functions will get done way faster. And then, in addition to that, a huge quality increase. So, our research team went and actually benchmarked a lot of enterprise tasks for document extraction or

3:06classification against these functions, um and we're seeing 90 plus percent accuracy uh at uh across these benchmarks for for some of these new releases. So, the TLDR is we have new functions, we have much higher quality on these functions, and then also these functions are even faster. >> And if I'm not wrong, AI parse is the function behind um document extraction on Agent Brick.

3:28>> Yeah, yeah. So, if you go into like our agents tab, you'll see a few different options, document parsing, information extraction, text classification. Um and then, all three of them, if you give us like a PDF, image, file, whatever it might be, we'll leverage AI parse document as the first step before it does the ultimate extraction or classification. So, you

3:50kind of also have that like silver layer if you want to reuse that for like other use cases of your document before you do your ultimate downstream task on it. >> I've seen many questions and by the way, make sure to join the Databricks subreddit. It's easy Databricks r/databricks. So, join the subreddit where you can ask all your questions and

4:12raise discussions about products. Now, I've seen one of the questions was about what's the difference between using AI query and the AI functions. >> Yeah. Yeah, that's a that's a great question. So, AI query is awesome. It It basically lets you give us any LLM and any prompt and we can basically run batch inference on your data

4:34to process your data at scale. But, what we really really honed in on when building these AI functions was getting our research and engineering and development teams to kind of push the Pareto curve or the Pareto frontier on how high of quality we can get you when we optimize for specific task at much lower cost or throughput than if you were to go and kind of try yourself with

4:57with the LLM and a DIY approach. So, our goal is really to like enable these workflows at scale. So, we were really focused on basically building the highest quality research systems that were tailored very specifically to document parsing, to classification and extraction in a way that allowed us to then also give you the optimal cost and the optimal throughput on those use

5:19cases. Because LLMs today, they can do anything. They can code for you, they can write stories for you. But, if you want something that's just really good at document extraction, there's a huge benefit, especially at scale, of using an optimized system behind the hood as opposed to just kind of winging it yourself. So, that's one of our our main goals is to really push the boundaries on what was possible and what kind of experience we could deliver to customers

5:42trying to process their data at really really large volume. So, millions of records a day, for example. >> That's amazing. And I've seen many people using AI functions, but the same questions always came up was, "Okay, that's good. I can use them easily, but how much or how can I track the cost easily?" >> Yeah, that's a that's a great question. So, um our AI functions are all tracked in

6:04system tables. And so, especially for our new ones, you'll actually just see like product name AI functions, and then it'll tag which AI function was used and and give you the usage and the billing of that AI function for every every run or every job that you kind of run using that function. And it also kind of give you metadata on like who used it, what was like the job or run ID or query ID

6:26that leveraged it. And then that way you can attribute it back to specific pipelines, for example, that you might be running them in. >> And for those who don't know, you can also use the query tag to make sure that whenever you run a query, you have a query tag, and you'll be able to get the cost based on on the query tag and aggregate all those costs, for example, for the same query that runs maybe every day or or every week. And I guess,

6:49Archika, you have prepared a demo? >> Yeah, yeah. I'm super excited to show you what we've been working on. So, we do have a demo. Awesome. So, what you should be seeing right now is SharePoint. This is where, you know, a lot of our customers' enterprise data and documents might live. So, kind of want to start here for our demo and show you how easy it is to get this data into Databricks and then process it and

7:11structure that data with our with our AI functions for downstream use cases. And so, basically here, I want to collect a variety of like different kinds of documents that an enterprise might be dealing with on any given day. We have some contracts here, real contracts from like SEC filings. Over here, we have some bank statements

7:35as well. Over here, pretty messy, complex tables. And then we even have some electronic invoices here. And so, our goal today is how can we turn all these messy documents into something that my business can actually make decisions on, automate workflows based off of, and ultimately use to like drive more outcomes, whether that's like revenue,

7:58business decisions, whatever it might be. Okay, cool. So, now that I have all these documents, I'm going to actually go in and show you how easy it is today to get those documents into Databricks with Lakehouse Connect and and start using them from day zero. So, now I'm in Databricks and what you see here is really just one cell of code. It's really really easy.

8:21I'm pointing my Lakehouse Connect for SharePoint specifically to the SharePoint URL, so the SharePoint site that I was just on. I've set up a Unity Catalog connection already in Databricks to securely authenticate to the SharePoint site. And then if I zoom in here, I also I'm going to write this to like a target table. And ultimately, I'm just going to be reading all the files from SharePoint

8:45here into Databricks. And as I like put this in a pipeline, I can incrementally basically read these files in as they come in, as new files come in, as old files change, and keep that system of record in Databricks up to date using our managed connectors. And so, if I run this cell real quick, I'm going to show you what you're going to get. Cool. Okay, so as you can see,

9:09I've pretty much ingested all the files that we saw in SharePoint. So, we have some electronic invoices here, we have the contracts here, and then we have the bank statements. So, now all this data lives in Databricks in a table with a binary representing every single file, and I can use that super easily downstream now for any of my AI functions. So, maybe let's do step

9:33one and let's parse these files to actually see like what text, tables, images actually live within each of these files. So, in order to do that, I'm going to go to the agents page in Databricks and switch over here. So, I am going to zoom in real quick so we can see a little bit better. Um I'm in the agents tab of Databricks right over here

9:56and I am going to go and click on create agent up here and I have a few different options. I'm going to filter to functions cuz that's what we're going to be using today. But, as you can see, I can do a few different things. I can go straight to extracting information, I can classify these documents, or I can parse them to like really understand what is the data that lives in these documents, OCR them, and then save them

10:19for downstream usage. So, I'm going to do document parsing to start off with. Um and it's really easy to like get your documents into here. I can upload them from a local um file finder, for example. I can select a volume that already exists in Databricks, but in this case, as you might remember, I have my documents in uh SharePoint uh produced table.

10:42I'll go to my schema here and let's see if we can find SharePoint. There we go. Okay, awesome. So, this is the table and I'll select the binary column that you kind of saw and I will start parsing. Awesome. So, what you can see is that the documents

11:05have successfully parsed. Um and we can actually see exactly like visually what we have stripped out of all of these documents and detected. Um in this parsing UI, you can really easily validate, okay, what are the elements that I detected with these clean bounding boxes around every single uh text or table that we extracted, and then we'll also kind of render the HTML

11:27structure of the table that we parsed out of your document. So, we have our invoices here, we have our contracts here. Um we have our bank statements here, and you can see how I did a pretty successful job of like pretty much capturing all of the information, uh whether it's like an image here with this figure description that you can see to the left, um the text or the tables

11:50from within these documents. So, this is really awesome because this makes it so much easier for your downstream agents to actually read the documents. Now that we have basically produced this data layer that um explains like exactly what lives in this document, what kind of elements those are, titles, text, tables, and and what the structure is. So, your agents don't need to directly look at these really messy PDF files that they struggle on. But, I don't want

12:12to stop here, actually. What I really care about is I want to extract like a few key entities that I care about from some of these documents. So, I'm going to actually try to do this with these bank statements, and as you might remember back when we were choosing our agent, we had a few different options, document parsing, information extraction, and classification. And I'm going to try to give the information extraction a whirl on some of these

12:34documents. So, I've already set up um an information extraction agent here. So, I'm going to switch over to that agent. Um and as you can see here, it basically is pointing to the three bank statement samples that I've been using all the way from SharePoint, where they lived in SharePoint, up until now. And I can natural in natural language kind of

12:57define what I want to extract. So, I'm going to tell my information extraction agent to extract customer name, primary account number, uh let's see, address, beginning balance, and ending balance. >> I love this part cuz once you're going to hit generate, it will it will detect

13:19the ski the right schema and then populate the populate the data. That's my favorite part. >> Let's see how it does. Um Perfect. Yeah, super snappy. So, um it's generated the actual schema. It's given it uh types as well and inferred what type it should be and then it also has generated a first pass that's um sensible descriptions for guiding the

13:43agent in terms of like what customer name or primary account number or address um actually means. And so, I'm going to save and run this extraction and see how it does on my bank statements. And having like a UI for this is super helpful for a lot of customers because documents are like inherently a visual thing. It's kind of hard to like look at

14:05the parsed output and really compare this side by side. So, that's why we want we wanted to make this like a native experience in Databricks so you can iterate um a lot faster. Awesome. Okay, so it seems like I've extracted a few things. Maybe we can click into here. Um so, on the left side, I have my parsed output. Kind of like you saw in the last step of like, "Okay, what text,

14:29images, and tables actually live in this document?" And then on the right side, we have the actual extracted schema. Um so, based off of this, I think this looks pretty reasonable. We have a fake customer name, Jane Customer here that's been parsed um a primary account number that looks pretty accurate. The address seems to map with what we have on the

14:51uh left side here or on the right side here. Um and then we have a beginning and ending balance uh that is pretty much on the nail. So, it looks like pretty easily we're kind of ready to start extracting these insights from these pretty um gnarly bank statements that were very complex, messy initially to look at, but we've

15:14cleaned this up pretty well um and and used our information extraction agent to do this. So, the last step here is kind of how do I actually operationalize this into a pipeline? >> Yeah. >> So, I'm going to like click use agent here and you can do something really awesome where you can create a Spark declarative pipeline immediately um from the uh the schema that you kind of iterated

15:36on and generated in the UI step. And so, I've built a Spark declarative pipeline off of these AI functions here that I love to share to kind of show like what the end-to-end story here looks like. And so, this is my document intelligence pipeline that operates over the statements, contracts, and invoices that we saw in the last step. Um and so,

15:58what I'm basically doing here is taking in the raw documents in this step and then I am parsing them and saving the parsed output again as that like silver layer that I can use downstream for any task, whether it's like search, extraction, um putting it in like a a knowledge assistant, etc. And then I'm actually classifying what kind of document it is

16:20using AI classify, which is some of our other one of our other AI functions, and based off of that classification, I'm extracting different schemas cuz I'm going to care about extracting something different from a bank statement versus a contract versus an invoice. Um and so, it branches out really well here um and I've pretty much been able to like extract all the values that I care about from from each kind of document.

16:44>> I think it's super super interesting to use SDP because it's incremental, which means whenever you add a new PDF, you don't need to each time process everything. It will only process the additional PDF or document that was added, which going to also save you cost, which means going to avoid reprocessing the same extraction and same extraction and parsing, which is

17:06which can be a little bit expensive if you do it every day on the same document. So, with SDP, it'll going to be incremental and it means going to be easy. >> Yeah. Yeah, exactly. Awesome. Um cool. So, that's kind of like a quick overview of how to turn those really messy documents into structured insights and then operationalize this as like an intelligent document processing pipeline

17:29that can pretty much um fuel whatever downstream analytics or agents you're building on top of it. And I have kind of a bonus of like, "Hey, like what if we were to put these tables into Genie?" Cuz we all know and love how powerful Genie is on on querying your data. And now that we've built some of these structured insights, it seems like the perfect next place to go. So, uh really quick, I made a Genie space. Um it's pointing to all of those tables

17:52that we've kind of generated in that pipeline that I showed you. And I'm just going to ask it a question that we already know the answer to, which is um "In her latest bank statement, cuz Jane customer might have many, um what is Jane's ending balance?" Same. >> The something important. Uh Archika used

18:14the agent mode, which is a mode where it enables the thinking and can look broadly compared to the chat mode where it's a direct question. And And you see it's very fast and accurate. Say like, "Jane's customer's ending balance in her last statement was 10,520 bucks." As of uh

18:36the 50th 50th 50th June 2020 20 23. >> Yeah. Yeah, awesome. Yeah, this is um exactly kind of what we saw in the last step, so we know it's correct. And it's kind of a really good way of showing that these the structured data makes all of these downstream agents even more powerful, faster to query those documents, um and accurate as well.

18:58>> Yeah, and I think can really say that uh the platform is making uh or helping the people building the pipelines quickly and in a very easy way without requiring a large technical background. As in you're a data analyst, you can just access like you can build the first part that Archika showed you where it was like ingesting

19:21the binary file and storing them as a Delta table. You could have used Genie code to do those steps or maybe just avoid using Genie code if you want or maybe just drag and drop the data straight to the volume, then parse those documents, extract the information and the clarity pipeline. You can also use

19:43Genie code to add the AI classify to say, "Okay, if it's a doc if it's a bank statement, add the I don't know, the name of the bank. If it's a bill, add the name of the company, the date and stuff." And you can the classification part and then you use the last part to extract information, save them on a Delta table and then use Genie to ask questions and once you're

20:06done, you can maybe open it on your mobile phone using uh the Genie. Thank you, Archika. You made I mean, you showed us how easy it was to build uh the pipeline. All right, now we have this amazing document intelligence in the platform. Thank you, Archika. >> Awesome. Yeah, thank you. >> In Lake House, live data sees Lake Flow streams to flow. Iceberg and Delta Lake,

20:30tables a glow. Unity Catalog governs Lake base O and T base strong. Genie takes to SQL MOSAIC AND KML PROLONGS. YOU sipping and co-hosting like the party years. Data dreams scale higher and higher next gen. Drop it.