Getting started with the Open Source Synthetic Data SDK

0:04Hey, today I have the pleasure to host John. Hi John. Hey Yousef, good to be here. I'm super super happy to have you. And can you introduce yourself to the community? Yeah, sure thing. Pleasure to be here. My name is John Sullivan. I lead the go to market team at a company called Mostly AI and we're an Austrian founded Sirib startup working on a a pretty big problem albeit a small team

0:27which is that the majority of the world's valuable data remains underutilized. It's difficult to access and share. Often it's proprietary. It's it's personal information about us, financial data, healthcare data, behavioral data. And even if you do get access to that data, it's difficult to um extract insights and value for a lot of us. So what we do at most AI is enable data access and data insights for

0:50everyone. And we do it through a combination of our open source SDK which I'm going to demo today and works really really nicely in data and also our data intelligence platform uh patent pending. Um excited about the conversation. Cheers. So I think you talked also about mos.ai which is an Australian uh company which is awesome and um and is there like how does your company operates?

1:13like is it you're just like building open-source technologies or you still have like other things to handle on your side? Yeah, it's a combination of open source and also commercial options. So on the open source side, our synthetic data SDK allows folks to take a real world tabular data set and train a generative model from scratch. It's based on our model architecture. It's

1:36not based on anyone else's. And then you take that we call a generator which is a combination of the model weights and the metadata and from that generator you can flexibly create privacy preserving synthetic data. Um so that's the core open- source option that we have. We also have a mock data option and then on the commercial side the enterprise offering is the mostly AI data intelligence platform. We obviously got

1:59inspiration from you folks on on the naming and um that's a full enterprise option. It has a excellent UI so that basically anyone can work with synthetic data train models and extract insights. We also have a a Python native assistant which makes it easy in natural language chatdesque conversations with your data. So a mix of open source and enterprise options. Yes, that's amazing. like data

2:21bricks they have their they have like the open all the open source technologies like spark mlflow delta lake apache iceberg unity catalog and that's are that's the core of everywhere like a big company you need to have this open source uh yeah it's huge it's important for transparency uh with the community as well so all of our code then is available in our GitHub repo

2:44um it's not a blackbox folks can understand the model architecture and understand why our model makes certain decisions So yeah, took inspiration from from Databricks and others and in various aspects. That's amazing. And you can can you tell us more about your uh your Databricks your sorry your SDK regarding the synthetic uh data like what are like the requirements and how

3:06can I start using this SDK? Yeah, cool. Of course. Um so it's available under a fully permissive Apache v2 license. Um so you can do whatever you like for as long as you like with it. um fully feature complete from a model training and a data generation perspective. So everything we make available in the enterprise platform you can also do through our Python syntax and we have a

3:28few core primitives um mostly train mostly generate mostly probe. Um so it's kind of a in a nutshell two steps. You train a generative model from scratch and from that model which we call a generator you flexibly create synthetic data. So that's in essence what we do. The synthetic data looks and feels and means the same as your real world data in terms of requirements. All you need is your your source data. So you take a

3:51tabular data set CSV park time series uh relational database and feed it into the model. It learns the joint probability distributions. So the patterns distributions of the real world data deletes the real world data because we're a privacy first company and then you're left with a model that can create statistically representative synthetic samples that are 100% private. So easier to access, easier to share, and easier

4:13to consume across uh your teams, but also externally with partners. Yeah, that's um that's amazing. And can you maybe show us a demo of how we can get started quickly? Would love to. Yeah, absolutely. So um let me share screen here. I'll do the entire screen. So Yousef, is that coming through okay for you? Yes. Excellent. So this is who we are. Data access and data insights for for everyone. And as I

4:37mentioned, a combination of our mostly data intelligence platform and also our open- source synthetic data SDK. So I'll focus on the SDK for the demo. This is our GitHub repo and you can find all the information needed to get started here. As I mentioned, there is a few key primitives the spine of the workflow. Um, and to get folks help or help folks get

5:00started, I should say, um, we have a guide here and some usage examples. um and also a cheat sheet and more advanced tutorials. So the cool thing is you can um by default create representative privacy safe synthetic data but you can also create better than real data depending on your use cases. So you can mold and manipulate um the data in synthetic form things like conditionally

5:23generate a subset of your uh customers. You can 10x 100x the synthetic data. You can rebalance certain attributes. You can change the temperature of the synthetic data to bias towards outliers. So all of that's available for folks. I'll give a quick shout out at this point to uh the mostly AI prize. I mentioned it to you before Yousef. We're running a competition uh to see who can generate the best synthetic data across both a single table setup um and a

5:47sequential data setup giving uh $100,000 in cash away uh to promote open data and open source. Uh the competition isn't exclusive to our SDK. You're welcome to use any open source tool, but I think it's pretty cool for a small series B startup to be um yeah run the competition like that for for the community's benefit. That said, I'll move into Databricks. It's a Python

6:10library, so you can pip install it in any Python native environment. The scenario I'm going to run through in about 5 minutes is we have a sensitive production data set as you can see on screen now. It's the US census income. It's uh customer data, demographic data, and typically um it's usually quite locked down. So when we work with customers, they'll have a

6:32production area and that data is proprietary, it's sensitive, it's difficult to access, it's difficult to share. So in the demo, I'm going to show the simple endto-end workflow of how you can use our SDK directly in Databricks. Again, it's Apache V2 license, so people can run with it from today and it's fully feature complete. a few simple simple steps. I've pip installed um MOSI

6:56already and restarted the Python kernel. So the first thing I'm going to do for the demo is um initialize the SDK in local mode. So you can run it in local mode to consume the resources you have available um on your LTAP or in Databricks wherever it is that you're pip installing our Python library or you can run it in client mode and will talk to our platform which is typically deployed within your own secure environment. So

7:19I've initialized the client in local mode and I'm going to read in that table I showed before. And um next I'm going to do uh uh training. So I'm going to mostly train and train a gener generator uh from scratch. Um this is going to print out um our endto-end pipe but also a training log. And while that's running, let me just explain a little

7:41bit about what's going on. So the model is going to um pull, analyze, and encode the training data. So the income data set that I showed before and it's going to learn the statistical patterns and distributions of that real world data and then create a model which we call a generator. You'll see in the configuration I've kept it quite simple for the purpose of

8:04a demo. The only thing I've done is cap the training time to 1 minute. So here with the model training already started that should finish after 60 seconds. There's lots of other things you can do. I mentioned before it's fully feature complete. So from a training and generation perspective, everything you can do in our enterprise platform, you can do with our open source SDK. That means you can uh invoke differential privacy. So we have built-in privacy

8:26mechanisms and by default all of the synthetic data is private. But if you need an additional mathematical guarantee on top, you can invoke differential privacy. You can also create multi-table and time series synthetic data. So you can configure your primary and foreign key. And you've got flexible flexible compute as well. So CPU, GPU, whatever your preference

8:48is. We also support text data. Um so tabular text, time series is all possible via the SDK. Um so what's happening here? You'll see uh we are almost finished with training the model and you can see that the training log the validation losses begin to flat flatten out. So even though I've capped the training time at 1 minute, it

9:10looks like the model has become saturated on this training data set which is about 50,000 rows and 15 columns. So a ra rather small data set and now we are complete. So we've taken the real world data, we trained a model to understand the statistical patterns and distributions and it's left us with an artifact which we call a generator. Again that's a combination of the

9:32metadata and the model weights. And um what's pretty cool is that you can generate a generate a a full HTML report directly in your your notebook. Um and that's really crucial because synthetic data is still while it's very very powerful technology, it's still rather niche. Um so the two questions we always get asked

9:54are how do you know it's accurate and how do you know it's private? Because ultimately it's a drop in replacement for your sensitive production data. So you need to understand and have trust that it's capable of doing the job. uh and fulfilling your use case. So in this model report here, you'll see there's an overall accuracy score of 95.4%. Um but there's also a bunch of statistical analysis to demonstrate that the generator has effectively learned

10:18the training data. So you can see there's a correlation matrix here. You've got your original data in gray, your synthetic in green. You have a breakdown of the uni variant distributions. So each of the columns you'll see the synthetic data tracks the um original data very closely. Same should go for the bary distributions. So the pair-wise relationships again the synthetic data in green mirrors the gray

10:43original data. You have an accuracy uh per column and also a matrix. We provide even cosine similarity for highdimensional spaces and we have our privacy metrics as well. So our distance metrics are plotted in the report. So that's a really important piece of the workflow because you're taking the training data um you have access to it

11:05which ultimately you want to give this generator to a colleague, a partner, another data consumer so that they can flexibly create synthetic data and they're going to ask is it accurate? Is it private? Basically can I trust it? And that's where the model report comes in. Yeah. And you don't you have all the statistics generated for you. So you don't need to do this on your behalf which can be time consuming. Yeah.

11:27Exactly. And with that being said when we talk to customers we always do encourage them to first off um evaluate the quality of synthetic data outside the platform. So we do have in the tutorials I showed earlier we have um train synthetic test reel. So we tell folks look create some synthetic data take it out of the platform or away from the SDK workflow and um train a model

11:52using the synthetic data and the input data the training data the real data have a hold out uh 20% from the original data and then compare the results. So we definitely encourage folks to try it out themselves. You don't have to rely on our model report. And then what happens is that's kind of the penny drop moment because people say this is this is my model. Um this is my experiment. I

12:16understand the parameters and I can see the quality carries over into synthetic data. And from there once it gets kind of productionized people relied in on the model report because they have understood from their own experiment that uh they can trust the accuracy and the privacy. Uh so once you have that report there's a few things you can do. Another one of our primitives is mostly probe. you can without actually having to create a synthetic data set, you can

12:39just probe the model for statistically representative of see safe synthetic samples. Um, and what's cool about these is so there's a 100 for example, let's try 100,000. In a matter of seconds, you can probe that model and get statistical uh statistically representative synthetic samples. And they're fully disposable. So I can keep generating and over and over again and probing I should say over and over again and I'm going to

13:02get new synthetic samples which is really really cool. Um and it gives you the ability to inspect and analyze the data on the fly. I mentioned earlier by default everything is highly representative and privacy safe. But the beauty of the generator and the beauty of synthetic data is that you can augment it. So you can mold and manipulate your data set. For example, one of the things you can do is conditionally generate a synthetic data

13:25set. So in this example, I'm going to feed in a seed file to that generator which understands my real world data and I'm going to fix two columns. I'm going to fix the uh sex and the age. So I want only 35year-old males in my data set. And what that gives you back then is a mixed real synthetic data set. So I can single in on a specific cohort of my

13:48customer base, which are 35-year-old males. um fix those attributes as real and then I can synthetically generate all of the rest of the attributes which may be deemed to be too sensitive in in production form. Have a really cool use case which we're promoting at the moment with Swiss Post and they use conditional generation via our open source SDK to fix the

14:11longitudinal and latitudinal columns in their data set at an area code level. So that was deemed to be sufficiently aggregated. But what they couldn't do was access the granular customer information. So they fixed the longitudinal, latitudinal, real geo coordinates of their customer database. And then they synthetically generated all of the customer attributes. Um, which is really cool because they went

14:33from the ability to um access 11% of their customers who had given consent to 100% of their customers in synthetic form. So that's a a nice little use case for conditional generation. And then we can move on and use our third primitive which is mostly generate. So I can then um actually create an artifact a

14:55synthetic data set that we can push back into um our catalog. And um for this example I'm just going to use a quick 10,00 examples or samples I should say. And uh as you can see we're generating that data and it'll ultimately be published here super quick. You can see from a an eye level inspection that it

15:18is highly representative of the original data. Um and you have all of the structure retained but also deeper the correlations and the statistical quality is represented as well and retained in synthetic form. Um so I have my data set um which I've generated from my trained model and now I'm just you can save it to the catalog and then you can start doing your basic stuff transforming

15:42cleaning or whatever you need to you need. Exactly. That's it in a nutshell. Um the thing I'd mention is uh obviously once you get into your um your catalog here um I have an example already. This is the synthetic version and I guess the end point is because it's privacy safe because there's no real world customer information available there. You can

16:05enable faster and safer data sharing and data access and that's the value that we bring uh as partners and I should have mentioned at the start obviously formal ISV partners to Databricks uh faster safer better access to sensitive data using a synthetic replica. So in this example here, I've given it access to BI, data scientists, etc. So that's it

16:27in a nutshell. That's the uh synthetic data SDK and that's how it uh operates inside of your Databricks environment. I've noticed it's very easy to use. You just need pip install and and you're and you're good to go and just learn some few APIs to generate the data and then you can do whatever you want and it's very easy, very fast and robust. I guess

16:49I think you no longer have excuse to start doing what you need to do. You can start just use the synthetic gener generic data and you're ready to go. Most important open source which means no fees are accurate except the compute fees of course. That's it in a nutshell. Um and going back to kind of our mission statement, we just want to broaden the access um to data such that

17:12people can extract insights and it's not there to replace data scientists or data literate folks. It's to give folks the ability to access data that they wouldn't normally have access to and then ideulate with it and brainstorm and think what can I do with this highquality data and folks then through the power of LLMs our assistant your assistant can ideulate and talk a natural language and create a use case

17:35and a workflow and a a more finished product to bring to your data scientists. So maybe do 80% of the work and say look I now want to get this into production can you help me with the last 10 15 20% of the task to operationalize things. So it increases the ability of a organization to innovate based on you know that valuable proprietary data that they have. Awesome. Thank you so much for John for this uh for this

17:57introduction to your Python SDK. Looking forward to play with it. Cheers. Appreciate the time Yousef and take care. Bye. I like.